2024-01-21T04:12:58.889 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-21T04:12:58.890 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711 --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} --name yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711/orig.config.yaml 2024-01-21T04:12:58.925 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T04:12:59.004 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711 branch: wip-yuri8-testing-2024-01-18-0823 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7523711' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2024-01-18-0823 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 0edf41a622739843d6b978b179ff3227b476dd9d valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: debuginfo: true flavor: default sha1: 0edf41a622739843d6b978b179ff3227b476dd9d thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2024-01-18-0823 sha1: 0edf41a622739843d6b978b179ff3227b476dd9d owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9599 sha1: 0edf41a622739843d6b978b179ff3227b476dd9d sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2024-01-18-0823 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0edf41a622739843d6b978b179ff3227b476dd9d targets: smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJHhHBSinmQEofBQijDsym1K+SkxvxwWk9w04OcLVikHqZ6Uhn4Og3fI/aWrXrLEWArQdCj/i62a0r94q3wei3Q= smithi144.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPS4pNmIUfUvxeA5qS72DYwNe/zOtn6/GnN+cuB4nt/rM9D0AzxuESufov7R1JEN2b2pqSV503uetyIVzM9KDoA= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-19_16:35:08 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-21T04:12:59.004 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa; will attempt to use it 2024-01-21T04:12:59.004 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks 2024-01-21T04:12:59.005 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-21T04:12:59.006 INFO:teuthology.task.internal:Checking packages... 2024-01-21T04:12:59.031 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0edf41a622739843d6b978b179ff3227b476dd9d' 2024-01-21T04:12:59.031 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-21T04:12:59.031 INFO:teuthology.packaging:ref: None 2024-01-21T04:12:59.031 INFO:teuthology.packaging:tag: None 2024-01-21T04:12:59.031 INFO:teuthology.packaging:branch: wip-yuri8-testing-2024-01-18-0823 2024-01-21T04:12:59.031 INFO:teuthology.packaging:sha1: 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T04:12:59.032 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2024-01-18-0823 2024-01-21T04:12:59.177 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-766.g0edf41a6 2024-01-21T04:12:59.178 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-21T04:12:59.184 INFO:teuthology.task.internal:no buildpackages task found 2024-01-21T04:12:59.184 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-21T04:12:59.201 INFO:teuthology.task.internal:Saving configuration 2024-01-21T04:12:59.214 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-21T04:12:59.219 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T04:12:59.241 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 04:07:59.658104', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJHhHBSinmQEofBQijDsym1K+SkxvxwWk9w04OcLVikHqZ6Uhn4Og3fI/aWrXrLEWArQdCj/i62a0r94q3wei3Q='} 2024-01-21T04:12:59.265 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 04:07:59.659951', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPS4pNmIUfUvxeA5qS72DYwNe/zOtn6/GnN+cuB4nt/rM9D0AzxuESufov7R1JEN2b2pqSV503uetyIVzM9KDoA='} 2024-01-21T04:12:59.265 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-21T04:12:59.271 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-01-21T04:12:59.271 INFO:teuthology.task.internal:roles: ubuntu@smithi144.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-01-21T04:12:59.271 INFO:teuthology.run_tasks:Running task console_log... 2024-01-21T04:12:59.341 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb168868ee0>, signals=[15]) 2024-01-21T04:12:59.342 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-21T04:12:59.348 INFO:teuthology.task.internal:Opening connections... 2024-01-21T04:12:59.348 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-01-21T04:12:59.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T04:12:59.437 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi144.front.sepia.ceph.com 2024-01-21T04:12:59.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T04:12:59.524 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-21T04:12:59.530 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-01-21T04:12:59.547 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-01-21T04:12:59.547 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-01-21T04:12:59.604 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2024-01-21T04:12:59.604 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-21T04:12:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T04:12:59.606 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-01-21T04:12:59.629 DEBUG:teuthology.orchestra.run.smithi144:> uname -m 2024-01-21T04:12:59.646 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2024-01-21T04:12:59.646 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2024-01-21T04:12:59.703 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="CentOS Stream" 2024-01-21T04:12:59.703 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="9" 2024-01-21T04:12:59.703 INFO:teuthology.orchestra.run.smithi144.stdout:ID="centos" 2024-01-21T04:12:59.703 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE="rhel fedora" 2024-01-21T04:12:59.703 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="9" 2024-01-21T04:12:59.704 INFO:teuthology.orchestra.run.smithi144.stdout:PLATFORM_ID="platform:el9" 2024-01-21T04:12:59.704 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-21T04:12:59.704 INFO:teuthology.orchestra.run.smithi144.stdout:ANSI_COLOR="0;31" 2024-01-21T04:12:59.704 INFO:teuthology.orchestra.run.smithi144.stdout:LOGO="fedora-logo-icon" 2024-01-21T04:12:59.704 INFO:teuthology.orchestra.run.smithi144.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-21T04:12:59.704 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://centos.org/" 2024-01-21T04:12:59.704 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-21T04:12:59.704 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-21T04:12:59.704 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-21T04:12:59.704 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2024-01-21T04:12:59.729 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-21T04:12:59.738 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-21T04:12:59.742 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-21T04:12:59.743 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-01-21T04:12:59.745 DEBUG:teuthology.orchestra.run.smithi144:> test '!' -e /home/ubuntu/cephtest 2024-01-21T04:12:59.761 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-21T04:12:59.767 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-21T04:12:59.767 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-01-21T04:12:59.802 DEBUG:teuthology.orchestra.run.smithi144:> test -z $(ls -A /var/lib/ceph) 2024-01-21T04:12:59.884 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-21T04:12:59.936 INFO:teuthology.run_tasks:Running task kernel... 2024-01-21T04:12:59.949 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-21T04:12:59.949 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-21T04:12:59.949 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T04:12:59.950 DEBUG:teuthology.orchestra.run.smithi144:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-21T04:12:59.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T04:12:59.966 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-01-21T04:12:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T04:12:59.967 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-01-21T04:13:00.023 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-361.el9.x86_64 2024-01-21T04:13:00.023 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.14.0-361.el9.x86_64 2024-01-21T04:13:00.023 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2024-01-21T04:13:00.024 INFO:teuthology.orchestra.run.smithi144.stdout:5.14.0-361.el9.x86_64 2024-01-21T04:13:00.025 INFO:teuthology.task.kernel:Running kernel on smithi144: 5.14.0-361.el9.x86_64 2024-01-21T04:13:00.025 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum install -y kernel 2024-01-21T04:13:02.295 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 9.7 MB/s | 6.4 MB 00:00 2024-01-21T04:13:02.338 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - AppStream 8.2 MB/s | 6.4 MB 00:00 2024-01-21T04:13:03.897 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-01-21T04:13:03.995 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-01-21T04:13:06.298 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-01-21T04:13:09.693 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 1.5 MB/s | 8.0 MB 00:05 2024-01-21T04:13:14.011 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 5.7 MB/s | 19 MB 00:03 2024-01-21T04:13:19.895 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 19 MB 00:12 2024-01-21T04:13:21.077 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 5.9 MB 00:03 2024-01-21T04:13:22.595 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-01-21T04:13:23.604 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2024-01-21T04:13:24.079 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 5.9 MB 00:00 2024-01-21T04:13:25.554 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 15 kB 00:00 2024-01-21T04:13:28.463 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-01-21T04:13:30.644 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-21T04:13:30.645 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-21T04:13:30.740 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:13:30.751 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:13:30.751 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-21T04:13:30.751 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:13:30.751 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:13:30.751 INFO:teuthology.orchestra.run.smithi069.stdout: kernel x86_64 5.14.0-409.el9 CentOS-BaseOS 6.0 M 2024-01-21T04:13:30.751 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-01-21T04:13:30.751 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core x86_64 5.14.0-409.el9 CentOS-BaseOS 21 M 2024-01-21T04:13:30.751 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules x86_64 5.14.0-409.el9 CentOS-BaseOS 39 M 2024-01-21T04:13:30.751 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core x86_64 5.14.0-409.el9 CentOS-BaseOS 33 M 2024-01-21T04:13:30.751 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:13:30.751 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:13:30.752 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:13:30.752 INFO:teuthology.orchestra.run.smithi069.stdout:Install 4 Packages 2024-01-21T04:13:30.752 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:13:30.752 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 99 M 2024-01-21T04:13:30.752 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 122 M 2024-01-21T04:13:30.752 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:13:31.531 INFO:teuthology.orchestra.run.smithi069.stdout:(1/4): kernel-5.14.0-409.el9.x86_64.rpm 7.7 MB/s | 6.0 MB 00:00 2024-01-21T04:13:32.156 INFO:teuthology.orchestra.run.smithi069.stdout:(2/4): kernel-core-5.14.0-409.el9.x86_64.rpm 15 MB/s | 21 MB 00:01 2024-01-21T04:13:32.881 INFO:teuthology.orchestra.run.smithi069.stdout:(3/4): kernel-modules-5.14.0-409.el9.x86_64.rpm 18 MB/s | 39 MB 00:02 2024-01-21T04:13:33.340 INFO:teuthology.orchestra.run.smithi069.stdout:(4/4): kernel-modules-core-5.14.0-409.el9.x86_6 18 MB/s | 33 MB 00:01 2024-01-21T04:13:33.343 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:13:33.343 INFO:teuthology.orchestra.run.smithi069.stdout:Total 38 MB/s | 99 MB 00:02 2024-01-21T04:13:33.343 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:13:33.729 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:13:33.730 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:13:34.757 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:13:34.758 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:13:37.148 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:13:37.809 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-core-5.14.0-409.el9.x86_64 1/4 2024-01-21T04:13:37.940 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T04:13:40.692 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T04:13:40.874 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T04:13:45.586 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T04:13:45.769 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:13:50.490 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:13:57.193 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 656 kB/s | 20 MB 00:31 2024-01-21T04:14:02.018 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-01-21T04:14:04.182 INFO:teuthology.orchestra.run.smithi144.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-21T04:14:04.183 INFO:teuthology.orchestra.run.smithi144.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-21T04:14:04.271 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:14:04.279 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:14:04.279 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-01-21T04:14:04.279 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:14:04.279 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:14:04.280 INFO:teuthology.orchestra.run.smithi144.stdout: kernel x86_64 5.14.0-409.el9 CentOS-BaseOS 6.0 M 2024-01-21T04:14:04.280 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-01-21T04:14:04.280 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-core x86_64 5.14.0-409.el9 CentOS-BaseOS 21 M 2024-01-21T04:14:04.280 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules x86_64 5.14.0-409.el9 CentOS-BaseOS 39 M 2024-01-21T04:14:04.280 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-core x86_64 5.14.0-409.el9 CentOS-BaseOS 33 M 2024-01-21T04:14:04.280 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:14:04.280 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:14:04.280 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:14:04.280 INFO:teuthology.orchestra.run.smithi144.stdout:Install 4 Packages 2024-01-21T04:14:04.280 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:14:04.281 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 99 M 2024-01-21T04:14:04.281 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 122 M 2024-01-21T04:14:04.281 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:14:04.877 INFO:teuthology.orchestra.run.smithi144.stdout:(1/4): kernel-5.14.0-409.el9.x86_64.rpm 10 MB/s | 6.0 MB 00:00 2024-01-21T04:14:05.469 INFO:teuthology.orchestra.run.smithi144.stdout:(2/4): kernel-core-5.14.0-409.el9.x86_64.rpm 18 MB/s | 21 MB 00:01 2024-01-21T04:14:06.161 INFO:teuthology.orchestra.run.smithi144.stdout:(3/4): kernel-modules-5.14.0-409.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-01-21T04:14:06.628 INFO:teuthology.orchestra.run.smithi144.stdout:(4/4): kernel-modules-core-5.14.0-409.el9.x86_6 19 MB/s | 33 MB 00:01 2024-01-21T04:14:06.631 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:14:06.631 INFO:teuthology.orchestra.run.smithi144.stdout:Total 42 MB/s | 99 MB 00:02 2024-01-21T04:14:06.631 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:14:07.012 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:14:07.012 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:14:08.038 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:14:08.039 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:14:10.434 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:14:11.150 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-modules-core-5.14.0-409.el9.x86_64 1/4 2024-01-21T04:14:11.287 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T04:14:14.068 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T04:14:14.250 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T04:14:18.956 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T04:14:19.145 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:14:19.777 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:14:19.777 INFO:teuthology.orchestra.run.smithi069.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-21T04:14:19.777 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:14:19.846 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:14:20.281 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:14:20.281 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-5.14.0-409.el9.x86_64 1/4 2024-01-21T04:14:20.281 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T04:14:20.282 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T04:14:20.747 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:14:20.747 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:14:20.747 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:14:20.747 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-5.14.0-409.el9.x86_64 2024-01-21T04:14:20.747 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core-5.14.0-409.el9.x86_64 2024-01-21T04:14:20.747 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-5.14.0-409.el9.x86_64 2024-01-21T04:14:20.747 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core-5.14.0-409.el9.x86_64 2024-01-21T04:14:20.747 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:14:20.747 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:14:21.281 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T04:14:21.414 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-21T04:14:21.415 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-21T04:14:21.415 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-409.el9.x86_64 2024-01-21T04:14:21.415 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T04:14:21.415 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T04:14:21.415 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-21T04:14:21.415 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T04:14:21.415 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T04:14:21.523 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-21T04:14:21.524 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-21T04:14:21.524 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2024-01-21T04:14:21.557 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2024-01-21T04:14:21.557 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2024-01-21T04:14:21.557 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2024-01-21T04:14:21.558 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2024-01-21T04:14:21.559 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:14:21.559 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T04:14:21.559 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T04:14:21.559 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T04:14:21.559 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2024-01-21T04:14:21.559 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:14:21.559 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T04:14:21.560 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T04:14:21.560 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 5.14.0-409.el9.x86_64 2024-01-21T04:14:21.560 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T04:14:22.122 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2024-01-21T04:14:23.463 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-21T04:14:23.505 INFO:teuthology.orchestra.run.smithi069.stderr:done 2024-01-21T04:14:23.507 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-01-21T04:14:23.522 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.2aNgJxRr1G 2024-01-21T04:14:23.522 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2aNgJxRr1G 2024-01-21T04:14:23.590 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.2aNgJxRr1G 2024-01-21T04:14:23.732 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.2aNgJxRr1G is 6KB 2024-01-21T04:14:23.780 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.2aNgJxRr1G 2024-01-21T04:14:23.795 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T04:14:23.866 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-21T04:14:23.867 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-21T04:14:23.867 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-21T04:14:23.867 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64.conf 2024-01-21T04:14:23.868 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64 2024-01-21T04:14:23.869 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:14:24.014 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2024-01-21T04:14:53.140 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:14:53.141 INFO:teuthology.orchestra.run.smithi144.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-21T04:14:53.141 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:14:53.209 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:14:53.629 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:14:53.629 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-5.14.0-409.el9.x86_64 1/4 2024-01-21T04:14:53.629 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-21T04:14:53.629 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-21T04:14:54.017 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-21T04:14:54.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-21T04:14:54.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T04:14:54.040 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-21T04:14:54.040 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:14:54.040 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:14:54.040 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-5.14.0-409.el9.x86_64 2024-01-21T04:14:54.040 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-core-5.14.0-409.el9.x86_64 2024-01-21T04:14:54.040 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-5.14.0-409.el9.x86_64 2024-01-21T04:14:54.040 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-core-5.14.0-409.el9.x86_64 2024-01-21T04:14:54.040 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:14:54.040 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:14:54.563 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T04:14:54.733 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-21T04:14:54.733 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-21T04:14:54.733 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-409.el9.x86_64 2024-01-21T04:14:54.734 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-21T04:14:54.734 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-21T04:14:54.734 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-01-21T04:14:54.734 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi144.front.sepia.ceph.com, path=None, version=distro) 2024-01-21T04:14:54.734 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q kernel | sort -rV | head -n 1 2024-01-21T04:14:54.841 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-21T04:14:54.842 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-21T04:14:54.842 DEBUG:teuthology.orchestra.run.smithi144:> sudo rpm -qi grub2-tools 2024-01-21T04:14:54.883 INFO:teuthology.orchestra.run.smithi144.stdout:Name : grub2-tools 2024-01-21T04:14:54.883 INFO:teuthology.orchestra.run.smithi144.stdout:Epoch : 1 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Version : 2.06 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Release : 68.el9 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: x86_64 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Group : Unspecified 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Size : 8264254 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:License : GPLv3+ 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Packager : builder@centos.org 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:Vendor : CentOS 2024-01-21T04:14:54.884 INFO:teuthology.orchestra.run.smithi144.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-21T04:14:54.885 INFO:teuthology.orchestra.run.smithi144.stdout:Summary : Support tools for GRUB. 2024-01-21T04:14:54.885 INFO:teuthology.orchestra.run.smithi144.stdout:Description : 2024-01-21T04:14:54.885 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:14:54.885 INFO:teuthology.orchestra.run.smithi144.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-21T04:14:54.885 INFO:teuthology.orchestra.run.smithi144.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-21T04:14:54.885 INFO:teuthology.orchestra.run.smithi144.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-21T04:14:54.885 INFO:teuthology.orchestra.run.smithi144.stdout:hardware devices. 2024-01-21T04:14:54.885 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:14:54.885 INFO:teuthology.orchestra.run.smithi144.stdout:This subpackage provides tools for support of all platforms. 2024-01-21T04:14:54.886 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-21T04:14:54.886 INFO:teuthology.task.kernel:Updating grub on smithi144 to boot 5.14.0-409.el9.x86_64 2024-01-21T04:14:54.886 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-21T04:14:55.514 INFO:teuthology.orchestra.run.smithi144.stderr:Generating grub configuration file ... 2024-01-21T04:14:56.866 INFO:teuthology.orchestra.run.smithi144.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-21T04:14:56.901 INFO:teuthology.orchestra.run.smithi144.stderr:done 2024-01-21T04:14:56.903 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2024-01-21T04:14:56.918 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.uMMQpIrwpj 2024-01-21T04:14:56.919 DEBUG:teuthology.orchestra.run.smithi144:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uMMQpIrwpj 2024-01-21T04:14:56.989 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.uMMQpIrwpj 2024-01-21T04:14:57.129 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.uMMQpIrwpj is 6KB 2024-01-21T04:14:57.179 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.uMMQpIrwpj 2024-01-21T04:14:57.195 DEBUG:teuthology.orchestra.run.smithi144:> sudo /bin/ls /boot/loader/entries || true 2024-01-21T04:14:57.260 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-21T04:14:57.260 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-21T04:14:57.260 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-21T04:14:57.260 INFO:teuthology.orchestra.run.smithi144.stdout:15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64.conf 2024-01-21T04:14:57.261 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64 2024-01-21T04:14:57.413 DEBUG:teuthology.orchestra.run.smithi144:> sudo shutdown -r now 2024-01-21T04:15:12.444 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-01-21T04:15:21.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-21T04:15:21.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T04:15:24.512 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-01-21T04:15:27.419 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-01-21T04:15:27.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-01-21T04:15:27.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T04:15:36.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-21T04:15:36.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T04:15:45.980 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-01-21T04:15:54.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-01-21T04:15:54.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T04:15:58.080 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-01-21T04:16:08.362 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-01-21T04:16:08.805 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-21T04:16:08.806 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-409.el9.x86_64"... 2024-01-21T04:16:08.806 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-01-21T04:16:08.819 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-409.el9.x86_64 2024-01-21T04:16:08.820 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-409.el9.x86_64 vs 5.14.0-409.el9.x86_64 2024-01-21T04:16:08.820 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T04:16:08.820 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T04:16:09.821 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-21T04:16:09.821 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T04:16:09.916 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2024-01-21T04:16:09.949 DEBUG:teuthology.parallel:result is None 2024-01-21T04:16:10.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-01-21T04:16:10.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T04:16:41.878 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-01-21T04:16:42.292 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-01-21T04:16:42.293 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-409.el9.x86_64"... 2024-01-21T04:16:42.293 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-01-21T04:16:42.307 INFO:teuthology.orchestra.run.smithi144.stdout:5.14.0-409.el9.x86_64 2024-01-21T04:16:42.307 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-409.el9.x86_64 vs 5.14.0-409.el9.x86_64 2024-01-21T04:16:42.307 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-21T04:16:42.307 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-21T04:16:43.308 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-21T04:16:43.308 DEBUG:teuthology.orchestra.run.smithi144:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-21T04:16:43.381 INFO:teuthology.orchestra.run.smithi144.stdout:ttyS1 2024-01-21T04:16:43.421 DEBUG:teuthology.parallel:result is None 2024-01-21T04:16:43.422 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-21T04:16:43.428 INFO:teuthology.task.internal:Creating test directory... 2024-01-21T04:16:43.428 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T04:16:43.431 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-21T04:16:43.447 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-21T04:16:43.455 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-21T04:16:43.465 INFO:teuthology.task.internal:Creating archive directory... 2024-01-21T04:16:43.465 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T04:16:43.490 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-21T04:16:43.517 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-21T04:16:43.524 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-21T04:16:43.524 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T04:16:43.554 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-21T04:16:43.595 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T04:16:43.595 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T04:16:43.608 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T04:16:43.608 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-21T04:16:43.610 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-21T04:16:43.617 INFO:teuthology.task.internal:Configuring sudo... 2024-01-21T04:16:43.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T04:16:43.653 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-21T04:16:43.680 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-21T04:16:43.689 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-21T04:16:43.689 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T04:16:43.723 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-21T04:16:43.737 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T04:16:43.818 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T04:16:43.870 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T04:16:43.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T04:16:43.997 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:16:43.997 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T04:16:44.065 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T04:16:44.099 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-21T04:16:44.154 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T04:16:44.236 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-21T04:16:44.277 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:16:44.277 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-21T04:16:44.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-01-21T04:16:44.346 DEBUG:teuthology.orchestra.run.smithi144:> sudo service rsyslog restart 2024-01-21T04:16:44.393 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T04:16:44.427 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T04:16:44.775 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-21T04:16:44.783 INFO:teuthology.task.internal:Starting timer... 2024-01-21T04:16:44.783 INFO:teuthology.run_tasks:Running task pcp... 2024-01-21T04:16:44.792 INFO:teuthology.run_tasks:Running task selinux... 2024-01-21T04:16:44.800 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2024-01-21T04:16:44.857 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2024-01-21T04:16:44.859 DEBUG:teuthology.orchestra.run.smithi144:> sudo service auditd rotate 2024-01-21T04:16:44.929 INFO:teuthology.orchestra.run.smithi144.stdout:Rotating logs: 2024-01-21T04:16:44.931 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-21T04:16:44.931 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2024-01-21T04:16:44.953 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2024-01-21T04:16:44.953 DEBUG:teuthology.orchestra.run.smithi144:> /usr/sbin/getenforce 2024-01-21T04:16:44.981 INFO:teuthology.orchestra.run.smithi144.stdout:Permissive 2024-01-21T04:16:44.981 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi069.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi144.front.sepia.ceph.com': 'permissive'} 2024-01-21T04:16:44.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T04:16:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T04:16:45.022 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T04:16:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T04:16:45.051 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-21T04:16:45.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2024-01-21T04:16:45.097 DEBUG:teuthology.orchestra.run.smithi144:> sudo /usr/sbin/setenforce permissive 2024-01-21T04:16:45.133 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-21T04:16:45.142 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-21T04:16:45.160 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-21T04:16:45.162 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi069.front.sepia.ceph.com,smithi144.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-21T04:20:44.119 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi144.front.sepia.ceph.com')] 2024-01-21T04:20:44.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-21T04:20:44.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T04:20:44.209 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-01-21T04:20:44.282 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-01-21T04:20:44.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-01-21T04:20:44.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-21T04:20:44.368 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-01-21T04:20:44.442 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-01-21T04:20:44.443 INFO:teuthology.run_tasks:Running task clock... 2024-01-21T04:20:44.453 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-21T04:20:44.453 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T04:20:44.454 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T04:20:44.456 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-21T04:20:44.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T04:20:44.484 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T04:20:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T04:20:44.514 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-21T04:20:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2024-01-21T04:20:44.531 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-21T04:20:44.543 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2024-01-21T04:20:44.560 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T04:20:44.560 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: ntpd: command not found 2024-01-21T04:20:44.574 INFO:teuthology.orchestra.run.smithi144.stdout:506 Cannot talk to daemon 2024-01-21T04:20:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T04:20:44.590 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-21T04:20:44.606 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-21T04:20:44.625 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-01-21T04:20:44.629 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T04:20:44.629 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-01-21T04:20:44.630 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T04:20:44.630 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T04:20:44.630 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T04:20:44.630 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T04:20:44.661 INFO:teuthology.orchestra.run.smithi144.stderr:bash: line 1: ntpq: command not found 2024-01-21T04:20:44.664 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T04:20:44.664 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2024-01-21T04:20:44.664 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T04:20:44.665 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T04:20:44.665 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T04:20:44.665 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T04:20:44.665 INFO:teuthology.run_tasks:Running task install... 2024-01-21T04:20:44.673 DEBUG:teuthology.task.install:project ceph 2024-01-21T04:20:44.673 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'}} 2024-01-21T04:20:44.673 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'} 2024-01-21T04:20:44.673 INFO:teuthology.task.install:Using flavor: default 2024-01-21T04:20:44.679 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-01-21T04:20:44.679 INFO:teuthology.task.install:extra packages: [] 2024-01-21T04:20:44.679 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'tag': None, 'wait_for_package': False} 2024-01-21T04:20:44.680 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T04:20:44.682 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'tag': None, 'wait_for_package': False} 2024-01-21T04:20:44.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T04:20:44.831 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/ 2024-01-21T04:20:44.831 INFO:teuthology.task.install.rpm:Package version is 19.0.0-766.g0edf41a6 2024-01-21T04:20:44.901 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/ 2024-01-21T04:20:44.901 INFO:teuthology.task.install.rpm:Package version is 19.0.0-766.g0edf41a6 2024-01-21T04:20:45.001 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T04:20:45.001 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:20:45.002 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T04:20:45.030 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-01-21T04:20:45.030 DEBUG:teuthology.orchestra.run.smithi144:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0edf41a622739843d6b978b179ff3227b476dd9d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T04:20:45.071 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-01-18-0823/0edf41a622739843d6b978b179ff3227b476dd9d/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-21T04:20:45.071 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:20:45.071 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-21T04:20:45.099 DEBUG:teuthology.orchestra.run.smithi144:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T04:20:45.100 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-01-21T04:20:45.100 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0edf41a622739843d6b978b179ff3227b476dd9d/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-21T04:20:45.167 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-21T04:20:45.179 DEBUG:teuthology.orchestra.run.smithi144:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T04:20:45.245 INFO:teuthology.orchestra.run.smithi144.stdout:check_obsoletes = 1 2024-01-21T04:20:45.246 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-21T04:20:45.247 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum clean all 2024-01-21T04:20:45.313 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2024-01-21T04:20:45.315 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2024-01-21T04:20:45.487 INFO:teuthology.orchestra.run.smithi144.stdout:73 files removed 2024-01-21T04:20:45.515 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-radosgw 2024-01-21T04:20:45.559 INFO:teuthology.orchestra.run.smithi069.stdout:73 files removed 2024-01-21T04:20:45.586 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw 2024-01-21T04:20:46.264 INFO:teuthology.orchestra.run.smithi144.stdout:ceph packages for x86_64 186 kB/s | 80 kB 00:00 2024-01-21T04:20:46.289 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 207 kB/s | 80 kB 00:00 2024-01-21T04:20:46.606 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 36 kB/s | 9.5 kB 00:00 2024-01-21T04:20:46.698 INFO:teuthology.orchestra.run.smithi144.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2024-01-21T04:20:47.050 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-01-21T04:20:47.149 INFO:teuthology.orchestra.run.smithi144.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-01-21T04:20:47.492 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2024-01-21T04:20:47.559 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2024-01-21T04:20:48.901 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-01-21T04:20:48.926 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-01-21T04:20:50.659 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-01-21T04:20:51.350 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-01-21T04:20:54.160 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 19 MB 00:02 2024-01-21T04:20:59.302 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 5.9 MB 00:01 2024-01-21T04:21:00.853 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-01-21T04:21:02.150 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 19 MB 00:09 2024-01-21T04:21:02.357 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-01-21T04:21:07.092 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 5.9 MB 00:01 2024-01-21T04:21:07.221 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-01-21T04:21:09.126 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 15 kB 00:00 2024-01-21T04:21:09.740 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:21:09.753 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 8.6 M 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrading: 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout: librados2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.4 M 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.1 M 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.1 M 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 19 M 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 25 k 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-21T04:21:09.754 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 742 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 499 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 4.8 M 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 45 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 130 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 161 k 2024-01-21T04:21:09.755 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-21T04:21:09.756 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 321 k 2024-01-21T04:21:09.756 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 298 k 2024-01-21T04:21:09.756 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 99 k 2024-01-21T04:21:09.756 INFO:teuthology.orchestra.run.smithi144.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-21T04:21:09.756 INFO:teuthology.orchestra.run.smithi144.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-21T04:21:09.756 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:09.756 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:21:09.756 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:21:09.756 INFO:teuthology.orchestra.run.smithi144.stdout:Install 27 Packages 2024-01-21T04:21:09.756 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrade 2 Packages 2024-01-21T04:21:09.756 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:09.757 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 55 M 2024-01-21T04:21:09.757 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:21:10.445 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2024-01-21T04:21:11.081 INFO:teuthology.orchestra.run.smithi144.stdout:(1/29): ceph-base-19.0.0-766.g0edf41a6.el9.x86_ 5.6 MB/s | 5.1 MB 00:00 2024-01-21T04:21:11.131 INFO:teuthology.orchestra.run.smithi144.stdout:(2/29): ceph-selinux-19.0.0-766.g0edf41a6.el9.x 505 kB/s | 25 kB 00:00 2024-01-21T04:21:11.247 INFO:teuthology.orchestra.run.smithi144.stdout:(3/29): libcephfs2-19.0.0-766.g0edf41a6.el9.x86 6.2 MB/s | 742 kB 00:00 2024-01-21T04:21:11.373 INFO:teuthology.orchestra.run.smithi144.stdout:(4/29): ceph-radosgw-19.0.0-766.g0edf41a6.el9.x 7.2 MB/s | 8.6 MB 00:01 2024-01-21T04:21:11.406 INFO:teuthology.orchestra.run.smithi144.stdout:(5/29): libradosstriper1-19.0.0-766.g0edf41a6.e 3.1 MB/s | 499 kB 00:00 2024-01-21T04:21:11.465 INFO:teuthology.orchestra.run.smithi144.stdout:(6/29): python3-ceph-argparse-19.0.0-766.g0edf4 773 kB/s | 45 kB 00:00 2024-01-21T04:21:11.524 INFO:teuthology.orchestra.run.smithi144.stdout:(7/29): python3-ceph-common-19.0.0-766.g0edf41a 2.2 MB/s | 130 kB 00:00 2024-01-21T04:21:11.591 INFO:teuthology.orchestra.run.smithi144.stdout:(8/29): python3-cephfs-19.0.0-766.g0edf41a6.el9 2.4 MB/s | 161 kB 00:00 2024-01-21T04:21:11.658 INFO:teuthology.orchestra.run.smithi144.stdout:(9/29): python3-rados-19.0.0-766.g0edf41a6.el9. 4.7 MB/s | 321 kB 00:00 2024-01-21T04:21:11.733 INFO:teuthology.orchestra.run.smithi144.stdout:(10/29): librgw2-19.0.0-766.g0edf41a6.el9.x86_6 13 MB/s | 4.8 MB 00:00 2024-01-21T04:21:11.775 INFO:teuthology.orchestra.run.smithi144.stdout:(11/29): python3-rbd-19.0.0-766.g0edf41a6.el9.x 2.5 MB/s | 298 kB 00:00 2024-01-21T04:21:11.800 INFO:teuthology.orchestra.run.smithi144.stdout:(12/29): python3-rgw-19.0.0-766.g0edf41a6.el9.x 1.4 MB/s | 99 kB 00:00 2024-01-21T04:21:11.925 INFO:teuthology.orchestra.run.smithi144.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 705 kB/s | 106 kB 00:00 2024-01-21T04:21:11.951 INFO:teuthology.orchestra.run.smithi144.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-01-21T04:21:11.968 INFO:teuthology.orchestra.run.smithi144.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-01-21T04:21:12.035 INFO:teuthology.orchestra.run.smithi144.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-01-21T04:21:12.060 INFO:teuthology.orchestra.run.smithi144.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-01-21T04:21:12.077 INFO:teuthology.orchestra.run.smithi144.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 995 kB/s | 41 kB 00:00 2024-01-21T04:21:12.294 INFO:teuthology.orchestra.run.smithi144.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-01-21T04:21:12.352 INFO:teuthology.orchestra.run.smithi144.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-01-21T04:21:12.445 INFO:teuthology.orchestra.run.smithi144.stdout:(21/29): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-01-21T04:21:12.469 INFO:teuthology.orchestra.run.smithi144.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-01-21T04:21:12.504 INFO:teuthology.orchestra.run.smithi144.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-01-21T04:21:12.570 INFO:teuthology.orchestra.run.smithi144.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-01-21T04:21:12.613 INFO:teuthology.orchestra.run.smithi144.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.7 MB/s | 838 kB 00:00 2024-01-21T04:21:12.663 INFO:teuthology.orchestra.run.smithi144.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-01-21T04:21:13.147 INFO:teuthology.orchestra.run.smithi144.stdout:(27/29): librados2-19.0.0-766.g0edf41a6.el9.x86 6.3 MB/s | 3.4 MB 00:00 2024-01-21T04:21:13.338 INFO:teuthology.orchestra.run.smithi144.stdout:(28/29): ceph-common-19.0.0-766.g0edf41a6.el9.x 6.1 MB/s | 19 MB 00:03 2024-01-21T04:21:13.405 INFO:teuthology.orchestra.run.smithi144.stdout:(29/29): librbd1-19.0.0-766.g0edf41a6.el9.x86_6 4.2 MB/s | 3.1 MB 00:00 2024-01-21T04:21:13.410 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:21:13.410 INFO:teuthology.orchestra.run.smithi144.stdout:Total 15 MB/s | 55 MB 00:03 2024-01-21T04:21:13.410 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:21:13.446 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:21:13.446 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:21:13.901 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:21:13.902 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:21:15.241 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:21:15.338 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-21T04:21:15.361 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-01-21T04:21:15.368 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-01-21T04:21:15.427 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-01-21T04:21:15.751 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-01-21T04:21:15.817 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T04:21:15.908 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T04:21:15.996 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 6/31 2024-01-21T04:21:16.028 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T04:21:16.073 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T04:21:16.118 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-01-21T04:21:16.164 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x 9/31 2024-01-21T04:21:16.233 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 10/31 2024-01-21T04:21:16.257 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T04:21:16.314 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T04:21:16.421 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-21T04:21:16.465 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-21T04:21:16.504 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-21T04:21:16.537 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-21T04:21:16.566 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-21T04:21:16.599 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-21T04:21:16.908 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-21T04:21:16.937 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 19/31 2024-01-21T04:21:17.047 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 19/31 2024-01-21T04:21:17.075 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 20/31 2024-01-21T04:21:17.126 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-21T04:21:17.555 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-21T04:21:17.615 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 23/31 2024-01-21T04:21:17.668 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 23/31 2024-01-21T04:21:17.745 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 24/31 2024-01-21T04:21:17.783 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86 25/31 2024-01-21T04:21:17.844 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:21:17.847 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:21:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-21T04:21:17.847 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:21:17.847 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:21:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 8.6 M 2024-01-21T04:21:17.847 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2024-01-21T04:21:17.847 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.4 M 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.1 M 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.1 M 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 19 M 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 25 k 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 742 k 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-21T04:21:17.848 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 499 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 4.8 M 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 45 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 130 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 161 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 321 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 298 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 99 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-21T04:21:17.849 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-21T04:21:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:17.850 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:21:17.850 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:21:17.850 INFO:teuthology.orchestra.run.smithi069.stdout:Install 27 Packages 2024-01-21T04:21:17.850 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 2 Packages 2024-01-21T04:21:17.850 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:17.850 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 55 M 2024-01-21T04:21:17.850 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:21:18.462 INFO:teuthology.orchestra.run.smithi069.stdout:(1/29): ceph-base-19.0.0-766.g0edf41a6.el9.x86_ 9.6 MB/s | 5.1 MB 00:00 2024-01-21T04:21:18.504 INFO:teuthology.orchestra.run.smithi069.stdout:(2/29): ceph-selinux-19.0.0-766.g0edf41a6.el9.x 608 kB/s | 25 kB 00:00 2024-01-21T04:21:18.571 INFO:teuthology.orchestra.run.smithi069.stdout:(3/29): libcephfs2-19.0.0-766.g0edf41a6.el9.x86 11 MB/s | 742 kB 00:00 2024-01-21T04:21:18.629 INFO:teuthology.orchestra.run.smithi069.stdout:(4/29): libradosstriper1-19.0.0-766.g0edf41a6.e 8.4 MB/s | 499 kB 00:00 2024-01-21T04:21:18.913 INFO:teuthology.orchestra.run.smithi069.stdout:(5/29): librgw2-19.0.0-766.g0edf41a6.el9.x86_64 17 MB/s | 4.8 MB 00:00 2024-01-21T04:21:18.955 INFO:teuthology.orchestra.run.smithi069.stdout:(6/29): python3-ceph-argparse-19.0.0-766.g0edf4 1.1 MB/s | 45 kB 00:00 2024-01-21T04:21:19.014 INFO:teuthology.orchestra.run.smithi069.stdout:(7/29): python3-ceph-common-19.0.0-766.g0edf41a 2.2 MB/s | 130 kB 00:00 2024-01-21T04:21:19.081 INFO:teuthology.orchestra.run.smithi069.stdout:(8/29): python3-cephfs-19.0.0-766.g0edf41a6.el9 2.4 MB/s | 161 kB 00:00 2024-01-21T04:21:19.139 INFO:teuthology.orchestra.run.smithi069.stdout:(9/29): python3-rados-19.0.0-766.g0edf41a6.el9. 5.4 MB/s | 321 kB 00:00 2024-01-21T04:21:19.206 INFO:teuthology.orchestra.run.smithi069.stdout:(10/29): python3-rbd-19.0.0-766.g0edf41a6.el9.x 4.4 MB/s | 298 kB 00:00 2024-01-21T04:21:19.256 INFO:teuthology.orchestra.run.smithi069.stdout:(11/29): python3-rgw-19.0.0-766.g0edf41a6.el9.x 1.9 MB/s | 99 kB 00:00 2024-01-21T04:21:19.365 INFO:teuthology.orchestra.run.smithi069.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-01-21T04:21:19.407 INFO:teuthology.orchestra.run.smithi069.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-01-21T04:21:19.441 INFO:teuthology.orchestra.run.smithi069.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-01-21T04:21:19.499 INFO:teuthology.orchestra.run.smithi069.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-01-21T04:21:19.541 INFO:teuthology.orchestra.run.smithi069.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-01-21T04:21:19.575 INFO:teuthology.orchestra.run.smithi069.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-01-21T04:21:19.675 INFO:teuthology.orchestra.run.smithi069.stdout:(18/29): ceph-radosgw-19.0.0-766.g0edf41a6.el9. 4.9 MB/s | 8.6 MB 00:01 2024-01-21T04:21:19.900 INFO:teuthology.orchestra.run.smithi069.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 950 kB/s | 309 kB 00:00 2024-01-21T04:21:19.967 INFO:teuthology.orchestra.run.smithi069.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 390 kB/s | 26 kB 00:00 2024-01-21T04:21:19.988 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T04:21:20.030 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T04:21:20.034 INFO:teuthology.orchestra.run.smithi069.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-01-21T04:21:20.109 INFO:teuthology.orchestra.run.smithi069.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 899 kB/s | 67 kB 00:00 2024-01-21T04:21:20.202 INFO:teuthology.orchestra.run.smithi069.stdout:(23/29): libarrow-9.0.0-10.el9.x86_64.rpm 8.4 MB/s | 4.4 MB 00:00 2024-01-21T04:21:20.385 INFO:teuthology.orchestra.run.smithi069.stdout:(24/29): ceph-common-19.0.0-766.g0edf41a6.el9.x 7.9 MB/s | 19 MB 00:02 2024-01-21T04:21:20.410 INFO:teuthology.orchestra.run.smithi069.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 919 kB/s | 192 kB 00:00 2024-01-21T04:21:20.435 INFO:teuthology.orchestra.run.smithi069.stdout:(26/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-01-21T04:21:20.591 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T04:21:20.634 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 27/31 2024-01-21T04:21:20.719 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 27/31 2024-01-21T04:21:20.719 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-21T04:21:20.719 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-21T04:21:20.720 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:20.728 INFO:teuthology.orchestra.run.smithi069.stdout:(27/29): librados2-19.0.0-766.g0edf41a6.el9.x86 11 MB/s | 3.4 MB 00:00 2024-01-21T04:21:20.751 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T04:21:20.786 INFO:teuthology.orchestra.run.smithi069.stdout:(28/29): librbd1-19.0.0-766.g0edf41a6.el9.x86_6 8.9 MB/s | 3.1 MB 00:00 2024-01-21T04:21:20.828 INFO:teuthology.orchestra.run.smithi069.stdout:(29/29): thrift-0.15.0-2.el9.x86_64.rpm 3.6 MB/s | 1.6 MB 00:00 2024-01-21T04:21:20.830 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:21:20.831 INFO:teuthology.orchestra.run.smithi069.stdout:Total 19 MB/s | 55 MB 00:02 2024-01-21T04:21:20.831 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:21:20.864 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:21:20.864 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:21:21.329 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:21:21.329 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:21:22.692 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:21:22.773 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-21T04:21:22.808 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-01-21T04:21:22.858 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-01-21T04:21:23.222 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-01-21T04:21:23.265 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T04:21:23.357 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T04:21:23.443 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 6/31 2024-01-21T04:21:23.475 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T04:21:23.538 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T04:21:23.566 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-01-21T04:21:23.610 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x 9/31 2024-01-21T04:21:23.678 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 10/31 2024-01-21T04:21:23.702 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T04:21:23.777 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T04:21:23.882 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-21T04:21:23.926 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-21T04:21:23.966 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-21T04:21:24.008 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-21T04:21:24.037 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-21T04:21:24.069 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-21T04:21:24.377 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-21T04:21:24.407 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 19/31 2024-01-21T04:21:24.481 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 19/31 2024-01-21T04:21:24.512 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 20/31 2024-01-21T04:21:24.558 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-21T04:21:24.984 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-21T04:21:25.010 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 23/31 2024-01-21T04:21:25.081 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 23/31 2024-01-21T04:21:25.149 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 24/31 2024-01-21T04:21:25.178 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86 25/31 2024-01-21T04:21:27.396 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T04:21:27.425 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T04:21:27.950 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 26/31 2024-01-21T04:21:27.996 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 27/31 2024-01-21T04:21:28.083 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 27/31 2024-01-21T04:21:28.084 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-21T04:21:28.084 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-21T04:21:28.084 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:28.122 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T04:21:30.208 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T04:21:30.208 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /sys 2024-01-21T04:21:30.208 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /proc 2024-01-21T04:21:30.208 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /mnt 2024-01-21T04:21:30.208 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /var/tmp 2024-01-21T04:21:30.209 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /home 2024-01-21T04:21:30.209 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /root 2024-01-21T04:21:30.209 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /tmp 2024-01-21T04:21:30.209 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:31.008 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 29/31 2024-01-21T04:21:31.032 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 29/31 2024-01-21T04:21:31.032 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:21:31.032 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-21T04:21:31.032 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-21T04:21:31.032 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-21T04:21:31.032 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:31.034 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-21T04:21:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-21T04:21:31.119 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T04:21:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T04:21:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 1/31 2024-01-21T04:21:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2/31 2024-01-21T04:21:32.296 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 3/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 4/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 6/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x 8/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86 9/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 10/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 12/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 13/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-21T04:21:32.297 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T04:21:32.298 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-21T04:21:32.299 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 30/31 2024-01-21T04:21:33.217 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout:Upgraded: 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout: librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-21T04:21:33.218 INFO:teuthology.orchestra.run.smithi144.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.219 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-21T04:21:33.220 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.220 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.220 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:33.220 INFO:teuthology.orchestra.run.smithi144.stdout: re2-1:20211101-3.el9.x86_64 2024-01-21T04:21:33.220 INFO:teuthology.orchestra.run.smithi144.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-21T04:21:33.220 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:33.220 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:21:33.592 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-test 2024-01-21T04:21:34.142 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:27 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:21:34.298 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 46 M 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 164 k 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-21T04:21:34.299 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:34.300 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:21:34.300 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:21:34.300 INFO:teuthology.orchestra.run.smithi144.stdout:Install 6 Packages 2024-01-21T04:21:34.300 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:34.300 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 47 M 2024-01-21T04:21:34.300 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 201 M 2024-01-21T04:21:34.300 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:21:34.499 INFO:teuthology.orchestra.run.smithi144.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-01-21T04:21:34.549 INFO:teuthology.orchestra.run.smithi144.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-01-21T04:21:34.625 INFO:teuthology.orchestra.run.smithi144.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 2.4 MB/s | 186 kB 00:00 2024-01-21T04:21:34.692 INFO:teuthology.orchestra.run.smithi144.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-01-21T04:21:34.875 INFO:teuthology.orchestra.run.smithi144.stdout:(5/6): libcephsqlite-19.0.0-766.g0edf41a6.el9.x 285 kB/s | 164 kB 00:00 2024-01-21T04:21:37.528 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T04:21:37.528 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2024-01-21T04:21:37.528 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2024-01-21T04:21:37.528 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2024-01-21T04:21:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2024-01-21T04:21:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2024-01-21T04:21:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /root 2024-01-21T04:21:37.529 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2024-01-21T04:21:37.529 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:38.363 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 29/31 2024-01-21T04:21:38.387 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 29/31 2024-01-21T04:21:38.388 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:21:38.388 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-21T04:21:38.388 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-21T04:21:38.388 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-21T04:21:38.388 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:38.390 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-21T04:21:38.465 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-21T04:21:38.466 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T04:21:39.586 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T04:21:39.586 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 1/31 2024-01-21T04:21:39.586 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 3/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 4/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 5/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 6/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 7/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x 8/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86 9/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 10/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 11/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 12/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 13/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-21T04:21:39.587 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-21T04:21:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-21T04:21:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-21T04:21:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-21T04:21:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-21T04:21:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-21T04:21:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-21T04:21:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-21T04:21:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-21T04:21:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-21T04:21:39.588 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-21T04:21:39.589 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 28/31 2024-01-21T04:21:39.589 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-21T04:21:39.589 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 30/31 2024-01-21T04:21:40.655 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-21T04:21:40.655 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:40.655 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2024-01-21T04:21:40.655 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.655 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-21T04:21:40.656 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:40.657 INFO:teuthology.orchestra.run.smithi069.stdout: re2-1:20211101-3.el9.x86_64 2024-01-21T04:21:40.658 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-21T04:21:40.658 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:40.658 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:21:41.038 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-test 2024-01-21T04:21:41.599 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:26 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:21:41.758 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:21:41.758 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:21:41.759 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-21T04:21:41.759 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:21:41.759 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:21:41.759 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 46 M 2024-01-21T04:21:41.759 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-01-21T04:21:41.759 INFO:teuthology.orchestra.run.smithi069.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-21T04:21:41.759 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 164 k 2024-01-21T04:21:41.759 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-21T04:21:41.759 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-21T04:21:41.759 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-21T04:21:41.760 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:41.760 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:21:41.760 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:21:41.760 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2024-01-21T04:21:41.760 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:41.760 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 47 M 2024-01-21T04:21:41.761 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 201 M 2024-01-21T04:21:41.761 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:21:41.995 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-01-21T04:21:42.037 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-01-21T04:21:42.087 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-01-21T04:21:42.146 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-01-21T04:21:42.246 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): libcephsqlite-19.0.0-766.g0edf41a6.el9.x 338 kB/s | 164 kB 00:00 2024-01-21T04:21:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): ceph-test-19.0.0-766.g0edf41a6.el9.x86_6 6.9 MB/s | 46 MB 00:06 2024-01-21T04:21:48.407 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:21:48.408 INFO:teuthology.orchestra.run.smithi069.stdout:Total 7.0 MB/s | 47 MB 00:06 2024-01-21T04:21:48.408 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:21:48.427 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:21:48.428 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:21:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:21:48.801 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:21:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:21:49.443 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-21T04:21:49.478 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-21T04:21:49.530 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-21T04:21:49.580 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-21T04:21:49.624 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 5/6 2024-01-21T04:21:50.182 INFO:teuthology.orchestra.run.smithi144.stdout:(6/6): ceph-test-19.0.0-766.g0edf41a6.el9.x86_6 2.9 MB/s | 46 MB 00:15 2024-01-21T04:21:50.185 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:21:50.185 INFO:teuthology.orchestra.run.smithi144.stdout:Total 2.9 MB/s | 47 MB 00:15 2024-01-21T04:21:50.186 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:21:50.206 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:21:50.206 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:21:50.578 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:21:50.579 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:21:51.131 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:21:51.214 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-01-21T04:21:51.248 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-01-21T04:21:51.310 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-01-21T04:21:51.358 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-01-21T04:21:51.385 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 5/6 2024-01-21T04:21:53.974 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 5/6 2024-01-21T04:21:54.048 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 6/6 2024-01-21T04:21:55.899 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 5/6 2024-01-21T04:21:55.985 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 6/6 2024-01-21T04:21:56.504 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 6/6 2024-01-21T04:21:56.504 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 1/6 2024-01-21T04:21:56.504 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 2/6 2024-01-21T04:21:56.504 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-21T04:21:56.504 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-21T04:21:56.505 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-21T04:21:56.907 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-21T04:21:56.907 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:56.907 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:21:56.907 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:56.907 INFO:teuthology.orchestra.run.smithi069.stdout: jq-1.6-16.el9.x86_64 2024-01-21T04:21:56.908 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:56.908 INFO:teuthology.orchestra.run.smithi069.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-21T04:21:56.908 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-21T04:21:56.908 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-21T04:21:56.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:56.908 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:21:57.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph 2024-01-21T04:21:57.765 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:42 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:21:57.934 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:21:57.936 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-01-21T04:21:57.936 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-21T04:21:57.936 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 6.5 k 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 2.1 M 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.5 M 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 245 k 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 4.7 M 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 17 M 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-21T04:21:57.937 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-21T04:21:57.938 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================== 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout:Install 36 Packages 2024-01-21T04:21:57.939 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:21:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 30 M 2024-01-21T04:21:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 106 M 2024-01-21T04:21:57.940 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:21:58.393 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 6/6 2024-01-21T04:21:58.393 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 1/6 2024-01-21T04:21:58.393 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 2/6 2024-01-21T04:21:58.393 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-01-21T04:21:58.393 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-01-21T04:21:58.393 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-01-21T04:21:58.657 INFO:teuthology.orchestra.run.smithi069.stdout:(1/36): ceph-19.0.0-766.g0edf41a6.el9.x86_64.rp 36 kB/s | 6.5 kB 00:00 2024-01-21T04:21:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-01-21T04:21:58.831 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:58.832 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:21:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: jq-1.6-16.el9.x86_64 2024-01-21T04:21:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:21:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-21T04:21:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-21T04:21:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-21T04:21:58.832 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:58.832 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:21:58.840 INFO:teuthology.orchestra.run.smithi069.stdout:(2/36): ceph-mds-19.0.0-766.g0edf41a6.el9.x86_6 5.8 MB/s | 2.1 MB 00:00 2024-01-21T04:21:59.167 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph 2024-01-21T04:21:59.265 INFO:teuthology.orchestra.run.smithi069.stdout:(3/36): ceph-mon-19.0.0-766.g0edf41a6.el9.x86_6 7.7 MB/s | 4.7 MB 00:00 2024-01-21T04:21:59.349 INFO:teuthology.orchestra.run.smithi069.stdout:(4/36): ceph-mgr-modules-core-19.0.0-766.g0edf4 2.9 MB/s | 245 kB 00:00 2024-01-21T04:21:59.466 INFO:teuthology.orchestra.run.smithi069.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-01-21T04:21:59.499 INFO:teuthology.orchestra.run.smithi069.stdout:(6/36): ceph-mgr-19.0.0-766.g0edf41a6.el9.x86_6 1.5 MB/s | 1.5 MB 00:01 2024-01-21T04:21:59.525 INFO:teuthology.orchestra.run.smithi069.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-01-21T04:21:59.549 INFO:teuthology.orchestra.run.smithi069.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2024-01-21T04:21:59.592 INFO:teuthology.orchestra.run.smithi069.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 709 kB/s | 47 kB 00:00 2024-01-21T04:21:59.617 INFO:teuthology.orchestra.run.smithi069.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 578 kB/s | 39 kB 00:00 2024-01-21T04:21:59.651 INFO:teuthology.orchestra.run.smithi069.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-01-21T04:21:59.709 INFO:teuthology.orchestra.run.smithi069.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-01-21T04:21:59.734 INFO:teuthology.orchestra.run.smithi069.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-01-21T04:21:59.741 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:52 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:21:59.760 INFO:teuthology.orchestra.run.smithi069.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-01-21T04:21:59.785 INFO:teuthology.orchestra.run.smithi069.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-21T04:21:59.810 INFO:teuthology.orchestra.run.smithi069.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-01-21T04:21:59.905 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:21:59.908 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================== 2024-01-21T04:21:59.908 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-01-21T04:21:59.908 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================== 2024-01-21T04:21:59.908 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:21:59.908 INFO:teuthology.orchestra.run.smithi144.stdout: ceph x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 6.5 k 2024-01-21T04:21:59.908 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-01-21T04:21:59.908 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 2.1 M 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.5 M 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 245 k 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 4.7 M 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 17 M 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-21T04:21:59.909 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-21T04:21:59.910 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================== 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout:Install 36 Packages 2024-01-21T04:21:59.911 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:21:59.912 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 30 M 2024-01-21T04:21:59.912 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 106 M 2024-01-21T04:21:59.912 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:21:59.994 INFO:teuthology.orchestra.run.smithi069.stdout:(17/36): ceph-osd-19.0.0-766.g0edf41a6.el9.x86_ 14 MB/s | 17 MB 00:01 2024-01-21T04:22:00.152 INFO:teuthology.orchestra.run.smithi069.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 61 kB/s | 22 kB 00:00 2024-01-21T04:22:00.186 INFO:teuthology.orchestra.run.smithi069.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 227 kB/s | 43 kB 00:00 2024-01-21T04:22:00.211 INFO:teuthology.orchestra.run.smithi069.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 377 kB/s | 151 kB 00:00 2024-01-21T04:22:00.245 INFO:teuthology.orchestra.run.smithi069.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-01-21T04:22:00.270 INFO:teuthology.orchestra.run.smithi069.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-01-21T04:22:00.295 INFO:teuthology.orchestra.run.smithi069.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-01-21T04:22:00.321 INFO:teuthology.orchestra.run.smithi069.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-01-21T04:22:00.346 INFO:teuthology.orchestra.run.smithi069.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-01-21T04:22:00.371 INFO:teuthology.orchestra.run.smithi069.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-01-21T04:22:00.396 INFO:teuthology.orchestra.run.smithi069.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-01-21T04:22:00.422 INFO:teuthology.orchestra.run.smithi069.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 616 kB/s | 46 kB 00:00 2024-01-21T04:22:00.443 INFO:teuthology.orchestra.run.smithi144.stdout:(1/36): ceph-19.0.0-766.g0edf41a6.el9.x86_64.rp 29 kB/s | 6.5 kB 00:00 2024-01-21T04:22:00.480 INFO:teuthology.orchestra.run.smithi069.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 945 kB/s | 79 kB 00:00 2024-01-21T04:22:00.505 INFO:teuthology.orchestra.run.smithi069.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-01-21T04:22:00.531 INFO:teuthology.orchestra.run.smithi069.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-01-21T04:22:00.560 INFO:teuthology.orchestra.run.smithi144.stdout:(2/36): ceph-mgr-19.0.0-766.g0edf41a6.el9.x86_6 4.3 MB/s | 1.5 MB 00:00 2024-01-21T04:22:00.564 INFO:teuthology.orchestra.run.smithi069.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-01-21T04:22:00.590 INFO:teuthology.orchestra.run.smithi069.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 612 kB/s | 36 kB 00:00 2024-01-21T04:22:00.610 INFO:teuthology.orchestra.run.smithi144.stdout:(3/36): ceph-mds-19.0.0-766.g0edf41a6.el9.x86_6 5.4 MB/s | 2.1 MB 00:00 2024-01-21T04:22:00.640 INFO:teuthology.orchestra.run.smithi069.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-01-21T04:22:00.698 INFO:teuthology.orchestra.run.smithi069.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-01-21T04:22:00.711 INFO:teuthology.orchestra.run.smithi144.stdout:(4/36): ceph-mgr-modules-core-19.0.0-766.g0edf4 2.4 MB/s | 245 kB 00:00 2024-01-21T04:22:00.723 INFO:teuthology.orchestra.run.smithi069.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 149 kB/s | 20 kB 00:00 2024-01-21T04:22:00.728 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:00.728 INFO:teuthology.orchestra.run.smithi069.stdout:Total 11 MB/s | 30 MB 00:02 2024-01-21T04:22:00.746 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:00.786 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:00.786 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:00.828 INFO:teuthology.orchestra.run.smithi144.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-01-21T04:22:00.869 INFO:teuthology.orchestra.run.smithi144.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-01-21T04:22:00.895 INFO:teuthology.orchestra.run.smithi144.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-01-21T04:22:00.929 INFO:teuthology.orchestra.run.smithi144.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-01-21T04:22:00.954 INFO:teuthology.orchestra.run.smithi144.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-01-21T04:22:00.996 INFO:teuthology.orchestra.run.smithi144.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-01-21T04:22:01.063 INFO:teuthology.orchestra.run.smithi144.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-01-21T04:22:01.105 INFO:teuthology.orchestra.run.smithi144.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-01-21T04:22:01.138 INFO:teuthology.orchestra.run.smithi144.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-01-21T04:22:01.172 INFO:teuthology.orchestra.run.smithi144.stdout:(14/36): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2024-01-21T04:22:01.177 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:01.178 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:01.347 INFO:teuthology.orchestra.run.smithi144.stdout:(15/36): ceph-osd-19.0.0-766.g0edf41a6.el9.x86_ 21 MB/s | 17 MB 00:00 2024-01-21T04:22:01.372 INFO:teuthology.orchestra.run.smithi144.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 934 kB/s | 187 kB 00:00 2024-01-21T04:22:01.456 INFO:teuthology.orchestra.run.smithi144.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-01-21T04:22:01.598 INFO:teuthology.orchestra.run.smithi144.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 670 kB/s | 151 kB 00:00 2024-01-21T04:22:01.623 INFO:teuthology.orchestra.run.smithi144.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 260 kB/s | 43 kB 00:00 2024-01-21T04:22:01.674 INFO:teuthology.orchestra.run.smithi144.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-01-21T04:22:01.724 INFO:teuthology.orchestra.run.smithi144.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-01-21T04:22:01.774 INFO:teuthology.orchestra.run.smithi144.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-01-21T04:22:01.799 INFO:teuthology.orchestra.run.smithi144.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 236 kB/s | 18 kB 00:00 2024-01-21T04:22:01.850 INFO:teuthology.orchestra.run.smithi144.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-01-21T04:22:01.875 INFO:teuthology.orchestra.run.smithi144.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-01-21T04:22:01.903 INFO:teuthology.orchestra.run.smithi144.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-01-21T04:22:01.926 INFO:teuthology.orchestra.run.smithi144.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 916 kB/s | 46 kB 00:00 2024-01-21T04:22:01.951 INFO:teuthology.orchestra.run.smithi144.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-01-21T04:22:01.993 INFO:teuthology.orchestra.run.smithi144.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-01-21T04:22:02.010 INFO:teuthology.orchestra.run.smithi144.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-01-21T04:22:02.043 INFO:teuthology.orchestra.run.smithi144.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-01-21T04:22:02.060 INFO:teuthology.orchestra.run.smithi144.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-01-21T04:22:02.102 INFO:teuthology.orchestra.run.smithi144.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-01-21T04:22:02.128 INFO:teuthology.orchestra.run.smithi144.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.2 MB/s | 427 kB 00:00 2024-01-21T04:22:02.145 INFO:teuthology.orchestra.run.smithi144.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 477 kB/s | 20 kB 00:00 2024-01-21T04:22:02.245 INFO:teuthology.orchestra.run.smithi144.stdout:(36/36): ceph-mon-19.0.0-766.g0edf41a6.el9.x86_ 2.6 MB/s | 4.7 MB 00:01 2024-01-21T04:22:02.250 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:02.250 INFO:teuthology.orchestra.run.smithi144.stdout:Total 13 MB/s | 30 MB 00:02 2024-01-21T04:22:02.264 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:22:02.304 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:22:02.304 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:22:02.506 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:02.569 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-21T04:22:02.602 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-21T04:22:02.629 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-21T04:22:02.709 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-21T04:22:02.729 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:22:02.729 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:22:02.753 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-21T04:22:02.785 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-21T04:22:02.878 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-21T04:22:02.947 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-21T04:22:02.988 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-21T04:22:03.061 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-21T04:22:03.114 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-21T04:22:03.152 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-21T04:22:03.211 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-21T04:22:03.265 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-21T04:22:03.462 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-21T04:22:03.516 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-21T04:22:03.555 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-21T04:22:03.589 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-21T04:22:03.619 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-21T04:22:03.695 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-21T04:22:03.730 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-21T04:22:03.763 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-21T04:22:03.804 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-21T04:22:03.838 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-21T04:22:03.873 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-21T04:22:03.963 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-21T04:22:04.004 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-21T04:22:04.076 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-21T04:22:04.096 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:22:04.159 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-21T04:22:04.194 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-21T04:22:04.214 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-21T04:22:04.225 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-21T04:22:04.296 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-21T04:22:04.341 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-21T04:22:04.341 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-21T04:22:04.382 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-21T04:22:04.482 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-21T04:22:04.503 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.n 31/36 2024-01-21T04:22:04.533 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 32/36 2024-01-21T04:22:04.558 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 32/36 2024-01-21T04:22:04.558 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:04.558 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-21T04:22:04.558 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-21T04:22:04.558 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-21T04:22:04.559 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:04.565 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-21T04:22:04.606 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-21T04:22:04.687 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-21T04:22:04.748 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-21T04:22:04.787 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-21T04:22:04.836 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-21T04:22:04.891 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-21T04:22:05.092 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-21T04:22:05.142 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-21T04:22:05.183 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-21T04:22:05.215 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-21T04:22:05.245 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-21T04:22:05.322 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-21T04:22:05.355 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-21T04:22:05.397 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-21T04:22:05.438 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-21T04:22:05.472 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-21T04:22:05.507 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-21T04:22:05.597 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-21T04:22:05.639 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-21T04:22:05.711 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-21T04:22:05.859 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-21T04:22:05.975 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-21T04:22:05.992 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 33/36 2024-01-21T04:22:06.019 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 33/36 2024-01-21T04:22:06.020 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:06.020 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-21T04:22:06.020 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-21T04:22:06.020 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-21T04:22:06.020 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:06.142 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.n 31/36 2024-01-21T04:22:06.171 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 32/36 2024-01-21T04:22:06.196 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 32/36 2024-01-21T04:22:06.196 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:06.196 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-21T04:22:06.196 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-21T04:22:06.196 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-21T04:22:06.196 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:06.460 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 34/36 2024-01-21T04:22:06.483 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 34/36 2024-01-21T04:22:06.483 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:06.483 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-21T04:22:06.483 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-21T04:22:06.483 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-21T04:22:06.483 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:06.693 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 35/36 2024-01-21T04:22:06.716 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 35/36 2024-01-21T04:22:06.716 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:06.716 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-21T04:22:06.717 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-21T04:22:06.717 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-21T04:22:06.717 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:06.783 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 36/36 2024-01-21T04:22:07.630 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 33/36 2024-01-21T04:22:07.657 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 33/36 2024-01-21T04:22:07.657 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:07.657 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-21T04:22:07.657 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-21T04:22:07.658 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-21T04:22:07.658 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:08.106 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 34/36 2024-01-21T04:22:08.129 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 34/36 2024-01-21T04:22:08.129 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:08.129 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-21T04:22:08.129 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-21T04:22:08.129 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-21T04:22:08.129 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:08.340 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 35/36 2024-01-21T04:22:08.363 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 35/36 2024-01-21T04:22:08.363 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:08.363 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-21T04:22:08.363 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-21T04:22:08.363 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-21T04:22:08.363 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:08.418 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 36/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 1/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 2/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 3/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 4/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 5/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.n 6/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-21T04:22:08.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-21T04:22:08.420 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-21T04:22:08.421 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-21T04:22:08.421 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-21T04:22:08.421 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-21T04:22:08.421 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-21T04:22:08.421 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-21T04:22:08.421 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-21T04:22:08.421 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-21T04:22:08.421 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-21T04:22:08.429 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 36/36 2024-01-21T04:22:09.276 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout: lua-5.4.4-4.el9.x86_64 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-21T04:22:09.279 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-21T04:22:09.280 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply-3.11-14.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:09.281 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:09.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base 2024-01-21T04:22:10.233 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:55 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:10.313 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 36/36 2024-01-21T04:22:10.313 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 1/36 2024-01-21T04:22:10.313 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 2/36 2024-01-21T04:22:10.313 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 3/36 2024-01-21T04:22:10.313 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 4/36 2024-01-21T04:22:10.313 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 5/36 2024-01-21T04:22:10.313 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.n 6/36 2024-01-21T04:22:10.313 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-21T04:22:10.313 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-21T04:22:10.314 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-21T04:22:10.314 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-21T04:22:10.314 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-21T04:22:10.314 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-21T04:22:10.314 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-21T04:22:10.314 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-21T04:22:10.314 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-21T04:22:10.314 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-21T04:22:10.314 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-21T04:22:10.315 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-21T04:22:10.315 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-21T04:22:10.316 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-21T04:22:10.316 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-21T04:22:10.316 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-21T04:22:10.316 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-21T04:22:10.316 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-21T04:22:10.316 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-21T04:22:10.317 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-21T04:22:10.317 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-21T04:22:10.317 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-21T04:22:10.317 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-21T04:22:10.318 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-21T04:22:10.318 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-21T04:22:10.318 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-21T04:22:10.318 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-21T04:22:10.318 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-21T04:22:10.318 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-21T04:22:10.333 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:10.383 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:10.384 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-01-21T04:22:10.384 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:10.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install cephadm 2024-01-21T04:22:11.000 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:55 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:11.131 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-21T04:22:11.131 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:11.131 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:22:11.131 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:11.131 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: lua-5.4.4-4.el9.x86_64 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-21T04:22:11.132 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ply-3.11-14.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-21T04:22:11.133 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-21T04:22:11.134 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-21T04:22:11.134 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-21T04:22:11.134 INFO:teuthology.orchestra.run.smithi144.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-21T04:22:11.134 INFO:teuthology.orchestra.run.smithi144.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-21T04:22:11.134 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-21T04:22:11.134 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-21T04:22:11.134 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-21T04:22:11.134 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:11.134 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:11.150 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:11.151 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:11.151 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-21T04:22:11.151 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:11.151 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:11.151 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 742 k 2024-01-21T04:22:11.151 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:11.151 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:11.151 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:11.151 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:22:11.151 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:11.152 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 742 k 2024-01-21T04:22:11.152 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 748 k 2024-01-21T04:22:11.152 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:11.393 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-base 2024-01-21T04:22:11.584 INFO:teuthology.orchestra.run.smithi069.stdout:cephadm-19.0.0-766.g0edf41a6.el9.noarch.rpm 1.7 MB/s | 742 kB 00:00 2024-01-21T04:22:11.584 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:11.585 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.7 MB/s | 742 kB 00:00 2024-01-21T04:22:11.585 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:11.589 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:11.590 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:11.595 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:11.595 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:11.671 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:11.956 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:04 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:12.055 INFO:teuthology.orchestra.run.smithi144.stdout:Package ceph-base-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:12.106 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:12.107 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-01-21T04:22:12.107 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:12.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install cephadm 2024-01-21T04:22:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:12.726 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:05 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:12.877 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:12.878 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:12.878 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-01-21T04:22:12.878 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:12.878 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:22:12.878 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 742 k 2024-01-21T04:22:12.878 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:12.878 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:22:12.879 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:12.879 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:22:12.879 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:12.879 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 742 k 2024-01-21T04:22:12.879 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 748 k 2024-01-21T04:22:12.879 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:22:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:12.964 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:12.964 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:13.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T04:22:13.715 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:00:58 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:13.753 INFO:teuthology.orchestra.run.smithi144.stdout:cephadm-19.0.0-766.g0edf41a6.el9.noarch.rpm 850 kB/s | 742 kB 00:00 2024-01-21T04:22:13.754 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:13.754 INFO:teuthology.orchestra.run.smithi144.stdout:Total 848 kB/s | 742 kB 00:00 2024-01-21T04:22:13.754 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:22:13.759 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:22:13.760 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:22:13.766 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:22:13.766 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:22:13.844 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:22:13.869 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:13.870 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:22:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:13.870 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 145 k 2024-01-21T04:22:13.870 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:13.870 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:13.871 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:22:13.871 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:13.871 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 145 k 2024-01-21T04:22:13.871 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 436 k 2024-01-21T04:22:13.871 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:14.089 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-immutable-object-cache-19.0.0-766.g0edf41a 668 kB/s | 145 kB 00:00 2024-01-21T04:22:14.089 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:14.090 INFO:teuthology.orchestra.run.smithi069.stdout:Total 662 kB/s | 145 kB 00:00 2024-01-21T04:22:14.090 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:14.097 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:14.097 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:14.148 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:14.149 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:14.286 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:14.340 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T04:22:14.360 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:14.363 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T04:22:14.364 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:14.364 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-21T04:22:14.364 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:14.436 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:14.820 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:15.112 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:22:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:15.112 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:15.112 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:15.327 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-immutable-object-cache 2024-01-21T04:22:15.388 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T04:22:15.389 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:15.389 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:15.389 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:15.389 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:15.389 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:15.615 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr 2024-01-21T04:22:15.900 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:08 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:16.055 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:16.056 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:16.056 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:22:16.057 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:16.057 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:22:16.057 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 145 k 2024-01-21T04:22:16.057 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:16.057 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:22:16.057 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:16.057 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:22:16.057 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:16.057 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 145 k 2024-01-21T04:22:16.057 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 436 k 2024-01-21T04:22:16.057 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:22:16.183 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:01 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:16.284 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:16.334 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:16.335 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-01-21T04:22:16.335 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:16.401 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T04:22:16.687 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-immutable-object-cache-19.0.0-766.g0edf41a 231 kB/s | 145 kB 00:00 2024-01-21T04:22:16.688 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:16.688 INFO:teuthology.orchestra.run.smithi144.stdout:Total 230 kB/s | 145 kB 00:00 2024-01-21T04:22:16.688 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:22:16.696 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:22:16.696 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:22:16.748 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:22:16.749 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:22:16.890 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:22:16.947 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:01 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:16.955 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T04:22:16.979 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T04:22:16.979 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:16.979 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-21T04:22:16.979 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:17.096 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:17.097 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:17.097 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-21T04:22:17.097 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:17.097 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:17.097 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 1.7 M 2024-01-21T04:22:17.097 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-01-21T04:22:17.097 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 28 k 2024-01-21T04:22:17.097 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 15 k 2024-01-21T04:22:17.097 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-21T04:22:17.098 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-21T04:22:17.098 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:17.098 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:17.098 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:17.098 INFO:teuthology.orchestra.run.smithi069.stdout:Install 5 Packages 2024-01-21T04:22:17.098 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:17.098 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 2.0 M 2024-01-21T04:22:17.099 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 8.5 M 2024-01-21T04:22:17.099 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:17.645 INFO:teuthology.orchestra.run.smithi069.stdout:(1/5): ceph-grafana-dashboards-19.0.0-766.g0edf 80 kB/s | 28 kB 00:00 2024-01-21T04:22:17.678 INFO:teuthology.orchestra.run.smithi069.stdout:(2/5): ceph-prometheus-alerts-19.0.0-766.g0edf4 39 kB/s | 15 kB 00:00 2024-01-21T04:22:17.905 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.e 1/1 2024-01-21T04:22:17.905 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:17.905 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:22:17.905 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:17.905 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:17.905 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:18.062 INFO:teuthology.orchestra.run.smithi069.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 74 kB/s | 31 kB 00:00 2024-01-21T04:22:18.147 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr 2024-01-21T04:22:18.329 INFO:teuthology.orchestra.run.smithi069.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 290 kB/s | 188 kB 00:00 2024-01-21T04:22:18.712 INFO:teuthology.orchestra.run.smithi069.stdout:(5/5): ceph-mgr-dashboard-19.0.0-766.g0edf41a6. 1.2 MB/s | 1.7 MB 00:01 2024-01-21T04:22:18.714 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:11 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:18.716 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:18.716 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-01-21T04:22:18.716 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:18.724 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:18.724 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:18.786 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:18.786 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:18.815 INFO:teuthology.orchestra.run.smithi144.stdout:Package ceph-mgr-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:18.864 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:18.865 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-01-21T04:22:18.866 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:18.938 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-dashboard 2024-01-21T04:22:19.032 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:19.117 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-21T04:22:19.169 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-21T04:22:19.205 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.no 3/5 2024-01-21T04:22:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.n 4/5 2024-01-21T04:22:19.473 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 5/5 2024-01-21T04:22:19.489 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:12 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:19.640 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:19.641 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:19.641 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-01-21T04:22:19.641 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:19.641 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:22:19.641 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 1.7 M 2024-01-21T04:22:19.641 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-01-21T04:22:19.641 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 28 k 2024-01-21T04:22:19.641 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 15 k 2024-01-21T04:22:19.641 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-21T04:22:19.641 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-21T04:22:19.642 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:19.642 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:22:19.642 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:19.642 INFO:teuthology.orchestra.run.smithi144.stdout:Install 5 Packages 2024-01-21T04:22:19.642 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:19.642 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 2.0 M 2024-01-21T04:22:19.642 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 8.5 M 2024-01-21T04:22:19.643 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:22:19.926 INFO:teuthology.orchestra.run.smithi144.stdout:(1/5): ceph-prometheus-alerts-19.0.0-766.g0edf4 78 kB/s | 15 kB 00:00 2024-01-21T04:22:20.026 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 5/5 2024-01-21T04:22:20.026 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.n 1/5 2024-01-21T04:22:20.026 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 2/5 2024-01-21T04:22:20.026 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.no 3/5 2024-01-21T04:22:20.027 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-21T04:22:20.027 INFO:teuthology.orchestra.run.smithi144.stdout:(2/5): ceph-grafana-dashboards-19.0.0-766.g0edf 97 kB/s | 28 kB 00:00 2024-01-21T04:22:20.085 INFO:teuthology.orchestra.run.smithi144.stdout:(3/5): ceph-mgr-dashboard-19.0.0-766.g0edf41a6. 5.0 MB/s | 1.7 MB 00:00 2024-01-21T04:22:20.210 INFO:teuthology.orchestra.run.smithi144.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 108 kB/s | 31 kB 00:00 2024-01-21T04:22:20.329 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-21T04:22:20.329 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:20.329 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:20.329 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:20.329 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:20.329 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:20.329 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-21T04:22:20.329 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-21T04:22:20.329 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:20.330 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:20.468 INFO:teuthology.orchestra.run.smithi144.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 426 kB/s | 188 kB 00:00 2024-01-21T04:22:20.470 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:20.471 INFO:teuthology.orchestra.run.smithi144.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-01-21T04:22:20.471 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:22:20.478 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:22:20.478 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:22:20.544 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:22:20.545 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:22:20.555 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T04:22:20.834 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:22:20.924 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-21T04:22:20.965 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-21T04:22:20.994 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.no 3/5 2024-01-21T04:22:21.107 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:06 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:21.231 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.n 4/5 2024-01-21T04:22:21.257 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout:=========================================================================================== 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout:=========================================================================================== 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 7.4 M 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-21T04:22:21.259 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-21T04:22:21.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-21T04:22:21.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-21T04:22:21.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-21T04:22:21.260 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-21T04:22:21.260 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:21.260 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:21.260 INFO:teuthology.orchestra.run.smithi069.stdout:=========================================================================================== 2024-01-21T04:22:21.260 INFO:teuthology.orchestra.run.smithi069.stdout:Install 12 Packages 2024-01-21T04:22:21.260 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:21.261 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 37 M 2024-01-21T04:22:21.261 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 209 M 2024-01-21T04:22:21.261 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:21.263 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 5/5 2024-01-21T04:22:21.493 INFO:teuthology.orchestra.run.smithi069.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 191 kB/s | 32 kB 00:00 2024-01-21T04:22:21.527 INFO:teuthology.orchestra.run.smithi069.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 517 kB/s | 17 kB 00:00 2024-01-21T04:22:21.560 INFO:teuthology.orchestra.run.smithi069.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-01-21T04:22:21.744 INFO:teuthology.orchestra.run.smithi069.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-01-21T04:22:21.808 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 5/5 2024-01-21T04:22:21.809 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.n 1/5 2024-01-21T04:22:21.809 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 2/5 2024-01-21T04:22:21.809 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.no 3/5 2024-01-21T04:22:21.809 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-21T04:22:21.878 INFO:teuthology.orchestra.run.smithi069.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-01-21T04:22:21.945 INFO:teuthology.orchestra.run.smithi069.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-01-21T04:22:21.970 INFO:teuthology.orchestra.run.smithi069.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-01-21T04:22:22.087 INFO:teuthology.orchestra.run.smithi069.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-76 9.8 MB/s | 7.4 MB 00:00 2024-01-21T04:22:22.110 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-21T04:22:22.110 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:22.110 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:22:22.110 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:22.110 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:22.110 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:22.110 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-21T04:22:22.110 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-21T04:22:22.110 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:22.111 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:22.137 INFO:teuthology.orchestra.run.smithi069.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 802 kB 00:00 2024-01-21T04:22:22.329 INFO:teuthology.orchestra.run.smithi069.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-01-21T04:22:22.379 INFO:teuthology.orchestra.run.smithi069.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 670 kB/s | 194 kB 00:00 2024-01-21T04:22:22.387 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-21T04:22:22.888 INFO:teuthology.orchestra.run.smithi069.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 328 kB/s | 245 kB 00:00 2024-01-21T04:22:22.891 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:22.892 INFO:teuthology.orchestra.run.smithi069.stdout:Total 23 MB/s | 37 MB 00:01 2024-01-21T04:22:22.905 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:22.929 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:22.929 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:22.979 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:15 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:23.140 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:23.141 INFO:teuthology.orchestra.run.smithi144.stdout:=========================================================================================== 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout:=========================================================================================== 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 7.4 M 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-21T04:22:23.142 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-21T04:22:23.143 INFO:teuthology.orchestra.run.smithi144.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-21T04:22:23.143 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-21T04:22:23.143 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-21T04:22:23.143 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-21T04:22:23.143 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:23.143 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:22:23.143 INFO:teuthology.orchestra.run.smithi144.stdout:=========================================================================================== 2024-01-21T04:22:23.143 INFO:teuthology.orchestra.run.smithi144.stdout:Install 12 Packages 2024-01-21T04:22:23.143 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:23.144 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 37 M 2024-01-21T04:22:23.144 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 209 M 2024-01-21T04:22:23.144 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:22:23.324 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:23.324 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:23.358 INFO:teuthology.orchestra.run.smithi144.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 223 kB/s | 32 kB 00:00 2024-01-21T04:22:23.400 INFO:teuthology.orchestra.run.smithi144.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 428 kB/s | 17 kB 00:00 2024-01-21T04:22:23.451 INFO:teuthology.orchestra.run.smithi144.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-01-21T04:22:23.501 INFO:teuthology.orchestra.run.smithi144.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 343 kB/s | 34 kB 00:00 2024-01-21T04:22:23.735 INFO:teuthology.orchestra.run.smithi144.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-01-21T04:22:23.818 INFO:teuthology.orchestra.run.smithi144.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-01-21T04:22:23.843 INFO:teuthology.orchestra.run.smithi144.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-01-21T04:22:24.019 INFO:teuthology.orchestra.run.smithi144.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 802 kB 00:00 2024-01-21T04:22:24.025 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:24.093 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-21T04:22:24.125 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-21T04:22:24.178 INFO:teuthology.orchestra.run.smithi144.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-01-21T04:22:24.202 INFO:teuthology.orchestra.run.smithi144.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-01-21T04:22:24.217 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-21T04:22:24.254 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-21T04:22:24.420 INFO:teuthology.orchestra.run.smithi144.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-01-21T04:22:24.433 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-21T04:22:24.469 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-21T04:22:24.545 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-21T04:22:25.208 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-21T04:22:25.322 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-21T04:22:26.559 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-21T04:22:26.628 INFO:teuthology.orchestra.run.smithi144.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-7 2.2 MB/s | 7.4 MB 00:03 2024-01-21T04:22:26.631 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:26.631 INFO:teuthology.orchestra.run.smithi144.stdout:Total 11 MB/s | 37 MB 00:03 2024-01-21T04:22:26.646 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:22:26.672 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:22:26.672 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:22:27.066 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:22:27.067 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:22:27.578 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-21T04:22:27.610 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 12/12 2024-01-21T04:22:27.776 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:22:27.834 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-21T04:22:27.874 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-21T04:22:27.974 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-21T04:22:28.027 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-21T04:22:28.213 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-21T04:22:28.251 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-21T04:22:28.327 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-21T04:22:28.972 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-21T04:22:29.080 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-21T04:22:30.171 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 12/12 2024-01-21T04:22:30.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 1/12 2024-01-21T04:22:30.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-21T04:22:30.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-21T04:22:30.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-21T04:22:30.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-21T04:22:30.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-21T04:22:30.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-21T04:22:30.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-21T04:22:30.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-21T04:22:30.172 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-21T04:22:30.173 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-21T04:22:30.317 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-21T04:22:30.623 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-21T04:22:30.623 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-21T04:22:30.624 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:30.625 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:30.970 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-rook 2024-01-21T04:22:31.339 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-21T04:22:31.370 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 12/12 2024-01-21T04:22:31.537 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:16 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:31.688 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:31.689 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-01-21T04:22:31.689 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-21T04:22:31.689 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-01-21T04:22:31.689 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:31.689 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 51 k 2024-01-21T04:22:31.689 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-01-21T04:22:31.689 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-21T04:22:31.689 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-21T04:22:31.689 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-21T04:22:31.689 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout:Install 13 Packages 2024-01-21T04:22:31.690 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:31.691 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 2.0 M 2024-01-21T04:22:31.691 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 26 M 2024-01-21T04:22:31.691 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:31.870 INFO:teuthology.orchestra.run.smithi069.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 141 kB/s | 18 kB 00:00 2024-01-21T04:22:31.895 INFO:teuthology.orchestra.run.smithi069.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 159 kB/s | 24 kB 00:00 2024-01-21T04:22:31.945 INFO:teuthology.orchestra.run.smithi069.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-01-21T04:22:31.970 INFO:teuthology.orchestra.run.smithi069.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-01-21T04:22:31.995 INFO:teuthology.orchestra.run.smithi069.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-01-21T04:22:32.023 INFO:teuthology.orchestra.run.smithi069.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-01-21T04:22:32.046 INFO:teuthology.orchestra.run.smithi069.stdout:(7/13): ceph-mgr-rook-19.0.0-766.g0edf41a6.el9. 166 kB/s | 51 kB 00:00 2024-01-21T04:22:32.105 INFO:teuthology.orchestra.run.smithi069.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 179 kB/s | 15 kB 00:00 2024-01-21T04:22:32.130 INFO:teuthology.orchestra.run.smithi069.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-01-21T04:22:32.188 INFO:teuthology.orchestra.run.smithi069.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-01-21T04:22:32.214 INFO:teuthology.orchestra.run.smithi069.stdout:(11/13): python3-google-auth-2.26.0-1.el9.noarc 1.2 MB/s | 202 kB 00:00 2024-01-21T04:22:32.247 INFO:teuthology.orchestra.run.smithi069.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-01-21T04:22:32.281 INFO:teuthology.orchestra.run.smithi069.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-01-21T04:22:32.283 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:32.283 INFO:teuthology.orchestra.run.smithi069.stdout:Total 3.5 MB/s | 2.0 MB 00:00 2024-01-21T04:22:32.283 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:32.296 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:32.296 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:32.391 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:32.391 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:32.940 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:33.094 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-21T04:22:33.149 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-21T04:22:33.203 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-21T04:22:33.242 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-21T04:22:33.283 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-21T04:22:33.375 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-21T04:22:33.493 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-21T04:22:33.542 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-21T04:22:33.973 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 12/12 2024-01-21T04:22:33.974 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41 1/12 2024-01-21T04:22:33.974 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-21T04:22:33.974 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-21T04:22:33.974 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-21T04:22:33.974 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-21T04:22:33.974 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-21T04:22:33.974 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-21T04:22:33.974 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-21T04:22:33.974 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-21T04:22:33.975 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-21T04:22:33.975 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-21T04:22:34.262 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-21T04:22:34.315 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-21T04:22:34.348 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-21T04:22:34.396 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-21T04:22:34.418 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 13/13 2024-01-21T04:22:34.461 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-21T04:22:34.462 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-21T04:22:34.463 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:34.463 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:34.817 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-rook 2024-01-21T04:22:35.220 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 13/13 2024-01-21T04:22:35.220 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 1/13 2024-01-21T04:22:35.220 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-21T04:22:35.220 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-21T04:22:35.220 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-21T04:22:35.220 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-21T04:22:35.220 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-21T04:22:35.220 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-21T04:22:35.221 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-21T04:22:35.221 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-21T04:22:35.221 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-21T04:22:35.221 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-21T04:22:35.221 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-21T04:22:35.385 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:28 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:35.537 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:35.538 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-01-21T04:22:35.538 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-01-21T04:22:35.538 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-01-21T04:22:35.538 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:22:35.538 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 51 k 2024-01-21T04:22:35.538 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-21T04:22:35.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-21T04:22:35.540 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:35.540 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:22:35.540 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-01-21T04:22:35.540 INFO:teuthology.orchestra.run.smithi144.stdout:Install 13 Packages 2024-01-21T04:22:35.540 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:35.540 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 2.0 M 2024-01-21T04:22:35.541 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 26 M 2024-01-21T04:22:35.541 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:22:35.595 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-21T04:22:35.595 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:35.595 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:35.595 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:35.595 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-21T04:22:35.595 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-21T04:22:35.595 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-21T04:22:35.596 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-21T04:22:35.596 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-21T04:22:35.596 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-21T04:22:35.596 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-21T04:22:35.596 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-21T04:22:35.596 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-21T04:22:35.596 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-21T04:22:35.596 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-21T04:22:35.596 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-21T04:22:35.596 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:35.596 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:35.745 INFO:teuthology.orchestra.run.smithi144.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 197 kB/s | 24 kB 00:00 2024-01-21T04:22:35.770 INFO:teuthology.orchestra.run.smithi144.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-01-21T04:22:35.821 INFO:teuthology.orchestra.run.smithi144.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-01-21T04:22:35.846 INFO:teuthology.orchestra.run.smithi144.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-01-21T04:22:35.857 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T04:22:35.871 INFO:teuthology.orchestra.run.smithi144.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-01-21T04:22:35.896 INFO:teuthology.orchestra.run.smithi144.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-01-21T04:22:35.947 INFO:teuthology.orchestra.run.smithi144.stdout:(7/13): ceph-mgr-rook-19.0.0-766.g0edf41a6.el9. 156 kB/s | 51 kB 00:00 2024-01-21T04:22:36.172 INFO:teuthology.orchestra.run.smithi144.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-01-21T04:22:36.197 INFO:teuthology.orchestra.run.smithi144.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-01-21T04:22:36.398 INFO:teuthology.orchestra.run.smithi144.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-01-21T04:22:36.417 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:21 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:36.456 INFO:teuthology.orchestra.run.smithi144.stdout:(11/13): python3-google-auth-2.26.0-1.el9.noarc 396 kB/s | 202 kB 00:00 2024-01-21T04:22:36.568 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-01-21T04:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-21T04:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-01-21T04:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 133 k 2024-01-21T04:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-01-21T04:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-21T04:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-21T04:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-21T04:22:36.569 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-21T04:22:36.570 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-21T04:22:36.570 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:36.570 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:36.570 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-01-21T04:22:36.570 INFO:teuthology.orchestra.run.smithi069.stdout:Install 6 Packages 2024-01-21T04:22:36.570 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:36.571 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 6.8 M 2024-01-21T04:22:36.571 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 33 M 2024-01-21T04:22:36.571 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:36.606 INFO:teuthology.orchestra.run.smithi144.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-01-21T04:22:36.674 INFO:teuthology.orchestra.run.smithi144.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 325 kB/s | 90 kB 00:00 2024-01-21T04:22:36.675 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:36.676 INFO:teuthology.orchestra.run.smithi144.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-01-21T04:22:36.676 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:22:36.689 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:22:36.689 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:22:36.788 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:22:36.789 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:22:36.799 INFO:teuthology.orchestra.run.smithi069.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-01-21T04:22:37.050 INFO:teuthology.orchestra.run.smithi069.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-01-21T04:22:37.092 INFO:teuthology.orchestra.run.smithi069.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-01-21T04:22:37.117 INFO:teuthology.orchestra.run.smithi069.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 871 kB/s | 58 kB 00:00 2024-01-21T04:22:37.151 INFO:teuthology.orchestra.run.smithi069.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 876 kB/s | 51 kB 00:00 2024-01-21T04:22:37.301 INFO:teuthology.orchestra.run.smithi069.stdout:(6/6): ceph-mgr-cephadm-19.0.0-766.g0edf41a6.el 198 kB/s | 133 kB 00:00 2024-01-21T04:22:37.304 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:37.304 INFO:teuthology.orchestra.run.smithi069.stdout:Total 9.2 MB/s | 6.8 MB 00:00 2024-01-21T04:22:37.305 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:37.314 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:37.314 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:37.337 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:22:37.403 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:37.403 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:37.511 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-21T04:22:37.566 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-21T04:22:37.622 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-21T04:22:37.659 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-21T04:22:37.690 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:37.701 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-21T04:22:37.795 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-21T04:22:37.859 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-21T04:22:37.903 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-21T04:22:37.911 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-21T04:22:37.960 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-21T04:22:38.426 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-21T04:22:38.505 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-21T04:22:38.566 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-21T04:22:38.590 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 6/6 2024-01-21T04:22:38.683 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-21T04:22:38.725 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-21T04:22:38.758 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-21T04:22:38.807 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-21T04:22:38.837 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 13/13 2024-01-21T04:22:39.424 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 6/6 2024-01-21T04:22:39.424 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/6 2024-01-21T04:22:39.424 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-21T04:22:39.424 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-21T04:22:39.424 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-21T04:22:39.424 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-21T04:22:39.569 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 13/13 2024-01-21T04:22:39.569 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 1/13 2024-01-21T04:22:39.569 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-21T04:22:39.569 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-21T04:22:39.569 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-21T04:22:39.569 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-21T04:22:39.569 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-21T04:22:39.569 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-21T04:22:39.569 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-21T04:22:39.570 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-21T04:22:39.570 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-21T04:22:39.570 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-21T04:22:39.570 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-21T04:22:39.752 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-21T04:22:39.752 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:39.752 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:39.752 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:39.753 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-21T04:22:39.753 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-21T04:22:39.753 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-21T04:22:39.753 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-21T04:22:39.753 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-21T04:22:39.753 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:39.753 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:39.958 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-21T04:22:39.959 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-21T04:22:39.960 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-21T04:22:39.960 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-21T04:22:39.960 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:39.960 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:39.983 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-fuse 2024-01-21T04:22:40.221 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-cephadm 2024-01-21T04:22:40.542 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:25 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:40.691 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:40.692 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:40.692 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-21T04:22:40.692 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:40.692 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:40.692 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 886 k 2024-01-21T04:22:40.692 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-01-21T04:22:40.692 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-21T04:22:40.692 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:40.693 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:40.693 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:40.693 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2024-01-21T04:22:40.693 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:40.693 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 964 k 2024-01-21T04:22:40.693 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.8 M 2024-01-21T04:22:40.693 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:40.790 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:33 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:40.942 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:40.942 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-01-21T04:22:40.942 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-01-21T04:22:40.942 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-01-21T04:22:40.942 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 133 k 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout:Install 6 Packages 2024-01-21T04:22:40.943 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:40.944 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 6.8 M 2024-01-21T04:22:40.944 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 33 M 2024-01-21T04:22:40.944 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:22:40.969 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 284 kB/s | 79 kB 00:00 2024-01-21T04:22:41.643 INFO:teuthology.orchestra.run.smithi144.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 630 kB/s | 228 kB 00:00 2024-01-21T04:22:41.678 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-fuse-19.0.0-766.g0edf41a6.el9.x86_6 899 kB/s | 886 kB 00:00 2024-01-21T04:22:41.679 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:41.680 INFO:teuthology.orchestra.run.smithi069.stdout:Total 977 kB/s | 964 kB 00:00 2024-01-21T04:22:41.680 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:41.701 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:41.702 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:41.779 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:41.779 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:41.877 INFO:teuthology.orchestra.run.smithi144.stdout:(2/6): ceph-mgr-cephadm-19.0.0-766.g0edf41a6.el 223 kB/s | 133 kB 00:00 2024-01-21T04:22:41.961 INFO:teuthology.orchestra.run.smithi144.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.5 MB/s | 5.8 MB 00:00 2024-01-21T04:22:41.979 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:42.104 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-21T04:22:42.111 INFO:teuthology.orchestra.run.smithi144.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 249 kB/s | 58 kB 00:00 2024-01-21T04:22:42.136 INFO:teuthology.orchestra.run.smithi144.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 291 kB/s | 51 kB 00:00 2024-01-21T04:22:42.161 INFO:teuthology.orchestra.run.smithi144.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-01-21T04:22:42.163 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:42.163 INFO:teuthology.orchestra.run.smithi144.stdout:Total 5.6 MB/s | 6.8 MB 00:01 2024-01-21T04:22:42.163 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:22:42.169 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T04:22:42.172 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:22:42.172 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:22:42.261 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:22:42.262 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:22:42.562 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:22:42.724 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-21T04:22:42.788 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-21T04:22:42.877 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T04:22:42.877 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/2 2024-01-21T04:22:43.204 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-21T04:22:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:43.205 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-21T04:22:43.205 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:43.205 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:43.300 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-21T04:22:43.391 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-21T04:22:43.459 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-21T04:22:43.475 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-volume 2024-01-21T04:22:43.492 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 6/6 2024-01-21T04:22:44.040 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:29 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:44.196 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 262 k 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 262 k 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.2 M 2024-01-21T04:22:44.197 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:44.296 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 6/6 2024-01-21T04:22:44.338 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 1/6 2024-01-21T04:22:44.338 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-21T04:22:44.338 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-21T04:22:44.338 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-21T04:22:44.338 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-21T04:22:44.449 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-volume-19.0.0-766.g0edf41a6.el9.noarch.rpm 1.0 MB/s | 262 kB 00:00 2024-01-21T04:22:44.449 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:44.449 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.0 MB/s | 262 kB 00:00 2024-01-21T04:22:44.450 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:44.458 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:44.458 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:44.496 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:44.496 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:44.600 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-21T04:22:44.600 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:44.600 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:22:44.600 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:44.601 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-21T04:22:44.601 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-21T04:22:44.601 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-21T04:22:44.601 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-21T04:22:44.601 INFO:teuthology.orchestra.run.smithi144.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-21T04:22:44.601 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:44.601 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:44.706 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:44.784 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:44.800 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:44.800 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:44.800 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-21T04:22:44.800 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:44.809 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-fuse 2024-01-21T04:22:45.378 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:38 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:45.533 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:45.533 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:45.533 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-01-21T04:22:45.534 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:45.534 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:22:45.534 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 886 k 2024-01-21T04:22:45.534 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-01-21T04:22:45.534 INFO:teuthology.orchestra.run.smithi144.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-21T04:22:45.534 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:45.534 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:22:45.534 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:45.534 INFO:teuthology.orchestra.run.smithi144.stdout:Install 2 Packages 2024-01-21T04:22:45.534 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:45.535 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 964 k 2024-01-21T04:22:45.535 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 2.8 M 2024-01-21T04:22:45.535 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:22:45.713 INFO:teuthology.orchestra.run.smithi144.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 442 kB/s | 79 kB 00:00 2024-01-21T04:22:45.903 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:45.904 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:45.904 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:45.904 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:45.904 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:45.904 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:46.147 INFO:teuthology.orchestra.run.smithi144.stdout:(2/2): ceph-fuse-19.0.0-766.g0edf41a6.el9.x86_6 1.4 MB/s | 886 kB 00:00 2024-01-21T04:22:46.148 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:46.148 INFO:teuthology.orchestra.run.smithi144.stdout:Total 1.5 MB/s | 964 kB 00:00 2024-01-21T04:22:46.148 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:22:46.168 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:22:46.169 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:22:46.188 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados-devel 2024-01-21T04:22:46.243 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:22:46.244 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:22:46.441 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:22:46.581 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-21T04:22:46.663 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T04:22:46.764 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:31 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:46.915 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:46.916 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:46.916 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:22:46.916 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:46.916 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:46.916 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 127 k 2024-01-21T04:22:46.916 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:46.917 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:46.917 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:46.917 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:22:46.917 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:46.917 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 127 k 2024-01-21T04:22:46.917 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 456 k 2024-01-21T04:22:46.917 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:47.361 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T04:22:47.362 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/2 2024-01-21T04:22:47.378 INFO:teuthology.orchestra.run.smithi069.stdout:librados-devel-19.0.0-766.g0edf41a6.el9.x86_64. 275 kB/s | 127 kB 00:00 2024-01-21T04:22:47.379 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:47.379 INFO:teuthology.orchestra.run.smithi069.stdout:Total 274 kB/s | 127 kB 00:00 2024-01-21T04:22:47.380 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:47.385 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:47.385 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:47.440 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:47.440 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:47.585 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:47.670 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-21T04:22:47.671 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:47.671 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:22:47.671 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-21T04:22:47.671 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:47.671 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:47.693 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:47.936 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-volume 2024-01-21T04:22:48.120 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:48.434 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:48.434 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:48.434 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:48.435 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:48.435 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:48.435 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:48.513 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:41 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:48.665 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:48.666 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:48.666 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2024-01-21T04:22:48.666 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:48.666 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:22:48.666 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume noarch 2:19.0.0-766.g0edf41a6.el9 ceph-noarch 262 k 2024-01-21T04:22:48.666 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:48.666 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:22:48.666 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:48.667 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:22:48.667 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:48.667 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 262 k 2024-01-21T04:22:48.667 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 1.2 M 2024-01-21T04:22:48.667 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:22:48.686 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs2 2024-01-21T04:22:49.248 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:34 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:49.351 INFO:teuthology.orchestra.run.smithi069.stdout:Package libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:49.360 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-volume-19.0.0-766.g0edf41a6.el9.noarch.rpm 379 kB/s | 262 kB 00:00 2024-01-21T04:22:49.360 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:49.361 INFO:teuthology.orchestra.run.smithi144.stdout:Total 378 kB/s | 262 kB 00:00 2024-01-21T04:22:49.361 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:22:49.369 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:22:49.369 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:22:49.403 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:49.404 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-01-21T04:22:49.404 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:49.408 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:22:49.409 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:22:49.475 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs-devel 2024-01-21T04:22:49.619 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:22:49.687 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:49.702 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:49.702 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:22:49.702 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-21T04:22:49.702 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:50.039 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:35 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:50.189 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:50.189 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 31 k 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 31 k 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 140 k 2024-01-21T04:22:50.190 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:50.557 INFO:teuthology.orchestra.run.smithi069.stdout:libcephfs-devel-19.0.0-766.g0edf41a6.el9.x86_64 85 kB/s | 31 kB 00:00 2024-01-21T04:22:50.557 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:50.557 INFO:teuthology.orchestra.run.smithi069.stdout:Total 85 kB/s | 31 kB 00:00 2024-01-21T04:22:50.558 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:50.561 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:50.561 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:50.590 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:50.591 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:50.676 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:50.787 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:50.814 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 1/1 2024-01-21T04:22:50.814 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:50.814 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:22:50.814 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume-2:19.0.0-766.g0edf41a6.el9.noarch 2024-01-21T04:22:50.815 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:50.815 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:51.124 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librados-devel 2024-01-21T04:22:51.170 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:51.449 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:51.449 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:51.450 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:51.450 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:51.450 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:51.450 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:51.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados2 2024-01-21T04:22:51.700 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:44 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:51.850 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:51.851 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:51.851 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:22:51.851 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:51.851 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:22:51.851 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 127 k 2024-01-21T04:22:51.851 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:51.851 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:22:51.852 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:51.852 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:22:51.852 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:51.852 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 127 k 2024-01-21T04:22:51.852 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 456 k 2024-01-21T04:22:51.852 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:22:52.179 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:37 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:52.280 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:52.324 INFO:teuthology.orchestra.run.smithi144.stdout:librados-devel-19.0.0-766.g0edf41a6.el9.x86_64. 269 kB/s | 127 kB 00:00 2024-01-21T04:22:52.324 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:52.325 INFO:teuthology.orchestra.run.smithi144.stdout:Total 268 kB/s | 127 kB 00:00 2024-01-21T04:22:52.325 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:22:52.330 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:22:52.331 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:22:52.333 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:52.334 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-01-21T04:22:52.334 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:52.385 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:22:52.386 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:22:52.405 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librbd1 2024-01-21T04:22:52.530 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:22:52.639 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:52.964 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:37 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:53.065 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:53.074 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:53.115 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:53.116 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-01-21T04:22:53.116 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:53.189 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rados 2024-01-21T04:22:53.381 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:53.381 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:53.381 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:22:53.381 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:53.381 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:53.381 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:53.628 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install libcephfs2 2024-01-21T04:22:53.739 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:38 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:53.839 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:53.889 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:53.890 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-01-21T04:22:53.890 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:53.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rgw 2024-01-21T04:22:54.209 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:47 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:54.319 INFO:teuthology.orchestra.run.smithi144.stdout:Package libcephfs2-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:54.370 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:54.371 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-01-21T04:22:54.371 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:54.444 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install libcephfs-devel 2024-01-21T04:22:54.515 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:39 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:54.615 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:54.665 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:54.666 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-01-21T04:22:54.666 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:54.734 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-cephfs 2024-01-21T04:22:55.010 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:48 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:55.160 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:22:55.160 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:22:55.161 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 31 k 2024-01-21T04:22:55.161 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:22:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:22:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:22:55.161 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 31 k 2024-01-21T04:22:55.161 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 140 k 2024-01-21T04:22:55.162 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:22:55.296 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:40 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:55.395 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:55.447 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:55.448 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-01-21T04:22:55.448 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:55.529 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rbd 2024-01-21T04:22:55.620 INFO:teuthology.orchestra.run.smithi144.stdout:libcephfs-devel-19.0.0-766.g0edf41a6.el9.x86_64 68 kB/s | 31 kB 00:00 2024-01-21T04:22:55.621 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:55.621 INFO:teuthology.orchestra.run.smithi144.stdout:Total 68 kB/s | 31 kB 00:00 2024-01-21T04:22:55.622 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:22:55.625 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:22:55.625 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:22:55.652 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:22:55.653 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:22:55.740 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:22:55.868 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:56.087 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:41 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:56.188 INFO:teuthology.orchestra.run.smithi069.stdout:Package python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:56.240 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:56.241 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-01-21T04:22:56.241 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:56.292 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:56.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-fuse 2024-01-21T04:22:56.553 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:56.554 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:56.554 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:22:56.554 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:56.554 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:22:56.554 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:56.734 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librados2 2024-01-21T04:22:56.870 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:41 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:57.022 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:57.022 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:57.022 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2024-01-21T04:22:57.023 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:57.023 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:57.023 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 85 k 2024-01-21T04:22:57.023 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:57.023 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:57.023 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:57.023 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:22:57.023 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:57.023 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 85 k 2024-01-21T04:22:57.023 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 231 k 2024-01-21T04:22:57.023 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:57.293 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:50 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:57.393 INFO:teuthology.orchestra.run.smithi144.stdout:Package librados2-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:57.444 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:57.445 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-01-21T04:22:57.445 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:57.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librbd1 2024-01-21T04:22:57.594 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-fuse-19.0.0-766.g0edf41a6.el9.x86_64.rpm 149 kB/s | 85 kB 00:00 2024-01-21T04:22:57.595 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:22:57.596 INFO:teuthology.orchestra.run.smithi069.stdout:Total 149 kB/s | 85 kB 00:00 2024-01-21T04:22:57.596 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:22:57.601 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:22:57.601 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:22:57.656 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:22:57.657 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:22:57.803 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:22:57.916 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:58.065 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:51 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:58.164 INFO:teuthology.orchestra.run.smithi144.stdout:Package librbd1-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:58.216 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:58.217 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-01-21T04:22:58.217 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:58.280 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rados 2024-01-21T04:22:58.320 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:58.636 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:22:58.637 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:58.637 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:22:58.637 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:22:58.637 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:58.637 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:22:58.839 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:51 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:58.859 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-mirror 2024-01-21T04:22:58.940 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-rados-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:58.990 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:58.991 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-01-21T04:22:58.991 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:59.060 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rgw 2024-01-21T04:22:59.445 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:44 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:22:59.603 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:22:59.604 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:59.604 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:22:59.604 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:59.604 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:22:59.604 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.1 M 2024-01-21T04:22:59.604 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:59.604 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:22:59.604 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:22:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:22:59.605 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:22:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 3.1 M 2024-01-21T04:22:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 13 M 2024-01-21T04:22:59.605 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:22:59.621 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:52 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:22:59.722 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-rgw-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:22:59.773 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:22:59.774 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-01-21T04:22:59.774 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:22:59.848 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-cephfs 2024-01-21T04:23:00.413 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:53 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:23:00.512 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-cephfs-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:23:00.563 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:23:00.564 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-01-21T04:23:00.564 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:23:00.635 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rbd 2024-01-21T04:23:01.194 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:54 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:23:01.293 INFO:teuthology.orchestra.run.smithi144.stdout:Package python3-rbd-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:23:01.344 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:23:01.345 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-01-21T04:23:01.345 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:23:01.405 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-fuse 2024-01-21T04:23:01.680 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-mirror-19.0.0-766.g0edf41a6.el9.x86_64.rpm 1.5 MB/s | 3.1 MB 00:02 2024-01-21T04:23:01.681 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:23:01.681 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.5 MB/s | 3.1 MB 00:02 2024-01-21T04:23:01.682 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:23:01.692 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:23:01.692 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:23:01.768 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:23:01.769 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:23:01.969 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:54 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:23:02.121 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:23:02.121 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:02.121 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2024-01-21T04:23:02.122 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:02.122 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:23:02.122 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 85 k 2024-01-21T04:23:02.122 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:02.122 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:23:02.122 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:02.122 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:23:02.122 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:02.122 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 85 k 2024-01-21T04:23:02.122 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 231 k 2024-01-21T04:23:02.122 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:23:02.204 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:23:02.264 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:02.289 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:02.289 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:23:02.289 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-21T04:23:02.290 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-21T04:23:02.290 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-21T04:23:02.290 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:02.515 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-fuse-19.0.0-766.g0edf41a6.el9.x86_64.rpm 217 kB/s | 85 kB 00:00 2024-01-21T04:23:02.516 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:23:02.516 INFO:teuthology.orchestra.run.smithi144.stdout:Total 216 kB/s | 85 kB 00:00 2024-01-21T04:23:02.516 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:23:02.523 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:23:02.523 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:23:02.578 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:23:02.578 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:23:02.715 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:23:02.814 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:03.202 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:03.478 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:03.478 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:03.478 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:23:03.478 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:23:03.478 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:03.478 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:23:03.504 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:03.504 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:03.504 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:23:03.504 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:23:03.504 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:03.504 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:23:03.742 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-nbd 2024-01-21T04:23:03.746 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-mirror 2024-01-21T04:23:04.320 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:49 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:23:04.322 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:01:57 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:23:04.470 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:23:04.471 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:23:04.471 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2024-01-21T04:23:04.471 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:23:04.471 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:23:04.471 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 170 k 2024-01-21T04:23:04.471 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:04.471 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:23:04.471 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:23:04.471 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:23:04.471 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:04.471 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 170 k 2024-01-21T04:23:04.472 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 490 k 2024-01-21T04:23:04.472 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:23:04.480 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:23:04.480 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:04.481 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:23:04.481 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:04.481 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:23:04.481 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 3.1 M 2024-01-21T04:23:04.481 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:04.481 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:23:04.481 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:04.481 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:23:04.481 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:04.482 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 3.1 M 2024-01-21T04:23:04.482 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 13 M 2024-01-21T04:23:04.482 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:23:05.226 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-nbd-19.0.0-766.g0edf41a6.el9.x86_64.rpm 226 kB/s | 170 kB 00:00 2024-01-21T04:23:05.226 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:23:05.226 INFO:teuthology.orchestra.run.smithi069.stdout:Total 225 kB/s | 170 kB 00:00 2024-01-21T04:23:05.226 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:23:05.234 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:23:05.235 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:23:05.295 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:23:05.296 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:23:05.441 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:23:05.540 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:05.948 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:06.267 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:06.268 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:06.268 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:23:06.268 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:23:06.268 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:06.268 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:23:06.500 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base-debuginfo 2024-01-21T04:23:06.685 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-mirror-19.0.0-766.g0edf41a6.el9.x86_64.rpm 1.4 MB/s | 3.1 MB 00:02 2024-01-21T04:23:06.686 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:23:06.686 INFO:teuthology.orchestra.run.smithi144.stdout:Total 1.4 MB/s | 3.1 MB 00:02 2024-01-21T04:23:06.687 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:23:06.697 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:23:06.697 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:23:06.774 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:23:06.774 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:23:07.065 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:01:52 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:23:07.207 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:23:07.222 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:23:07.222 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:23:07.223 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:23:07.223 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:23:07.223 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:23:07.223 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 67 M 2024-01-21T04:23:07.223 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-01-21T04:23:07.223 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 22 M 2024-01-21T04:23:07.223 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:07.223 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:23:07.223 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:23:07.223 INFO:teuthology.orchestra.run.smithi069.stdout:Install 2 Packages 2024-01-21T04:23:07.223 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:07.224 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 89 M 2024-01-21T04:23:07.224 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 490 M 2024-01-21T04:23:07.224 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:23:07.294 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:07.319 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:07.319 INFO:teuthology.orchestra.run.smithi144.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-21T04:23:07.319 INFO:teuthology.orchestra.run.smithi144.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-21T04:23:07.319 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-21T04:23:07.319 INFO:teuthology.orchestra.run.smithi144.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-21T04:23:07.319 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:08.521 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:08.522 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:08.522 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:23:08.522 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:23:08.522 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:08.522 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:23:08.796 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-nbd 2024-01-21T04:23:09.375 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:02 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:23:09.530 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 170 k 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 170 k 2024-01-21T04:23:09.531 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 490 k 2024-01-21T04:23:09.532 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:23:10.183 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-nbd-19.0.0-766.g0edf41a6.el9.x86_64.rpm 262 kB/s | 170 kB 00:00 2024-01-21T04:23:10.183 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:23:10.183 INFO:teuthology.orchestra.run.smithi144.stdout:Total 261 kB/s | 170 kB 00:00 2024-01-21T04:23:10.184 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:23:10.191 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:23:10.191 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:23:10.250 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:23:10.250 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:23:10.401 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:23:10.514 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:10.985 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:11.305 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:23:11.305 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:11.305 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:23:11.305 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:23:11.305 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:11.305 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:23:11.533 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-base-debuginfo 2024-01-21T04:23:12.102 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:05 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:23:12.257 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:23:12.258 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:12.258 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:23:12.258 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:12.258 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:23:12.258 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 67 M 2024-01-21T04:23:12.258 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-01-21T04:23:12.258 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 22 M 2024-01-21T04:23:12.258 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:12.258 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:23:12.258 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:12.259 INFO:teuthology.orchestra.run.smithi144.stdout:Install 2 Packages 2024-01-21T04:23:12.259 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:12.259 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 89 M 2024-01-21T04:23:12.259 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 490 M 2024-01-21T04:23:12.259 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:23:12.595 INFO:teuthology.orchestra.run.smithi069.stdout:(1/2): ceph-debuginfo-19.0.0-766.g0edf41a6.el9. 4.2 MB/s | 22 MB 00:05 2024-01-21T04:23:15.661 INFO:teuthology.orchestra.run.smithi144.stdout:(1/2): ceph-debuginfo-19.0.0-766.g0edf41a6.el9. 6.6 MB/s | 22 MB 00:03 2024-01-21T04:23:20.560 INFO:teuthology.orchestra.run.smithi144.stdout:(2/2): ceph-base-debuginfo-19.0.0-766.g0edf41a6 8.0 MB/s | 67 MB 00:08 2024-01-21T04:23:20.562 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:23:20.562 INFO:teuthology.orchestra.run.smithi144.stdout:Total 11 MB/s | 89 MB 00:08 2024-01-21T04:23:20.563 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:23:20.568 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:23:20.568 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:23:21.152 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:23:21.153 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:23:23.094 INFO:teuthology.orchestra.run.smithi069.stdout:(2/2): ceph-base-debuginfo-19.0.0-766.g0edf41a6 4.2 MB/s | 67 MB 00:15 2024-01-21T04:23:23.095 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:23:23.096 INFO:teuthology.orchestra.run.smithi069.stdout:Total 5.6 MB/s | 89 MB 00:15 2024-01-21T04:23:23.096 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:23:23.101 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:23:23.101 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:23:23.686 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:23:23.686 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:23:24.259 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:23:26.767 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:23:31.197 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/2 2024-01-21T04:23:31.277 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 2/2 2024-01-21T04:23:33.780 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/2 2024-01-21T04:23:33.911 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 2/2 2024-01-21T04:23:34.647 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 2/2 2024-01-21T04:23:34.647 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/2 2024-01-21T04:23:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T04:23:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:34.915 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:23:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:23:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:23:34.915 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:34.915 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:23:35.120 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-common-debuginfo 2024-01-21T04:23:35.689 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:28 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:23:35.846 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:23:35.847 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:35.847 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:23:35.847 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:35.847 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:23:35.847 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 411 M 2024-01-21T04:23:35.847 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:35.847 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:23:35.847 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:23:35.848 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:23:35.848 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:23:35.848 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 411 M 2024-01-21T04:23:35.848 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 1.7 G 2024-01-21T04:23:35.848 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:23:37.216 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 2/2 2024-01-21T04:23:37.216 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/2 2024-01-21T04:23:37.501 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2/2 2024-01-21T04:23:37.501 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:37.501 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:23:37.501 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:23:37.501 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:23:37.501 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:37.501 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:23:37.721 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-common-debuginfo 2024-01-21T04:23:38.299 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:02:23 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:23:38.460 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:23:38.460 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 411 M 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 411 M 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.7 G 2024-01-21T04:23:38.461 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:24:37.542 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-common-debuginfo-19.0.0-766.g0edf41a6.el9. 7.0 MB/s | 411 MB 00:59 2024-01-21T04:24:37.543 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:24:37.543 INFO:teuthology.orchestra.run.smithi069.stdout:Total 7.0 MB/s | 411 MB 00:59 2024-01-21T04:24:37.543 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:24:37.548 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:24:37.548 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:24:39.984 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:24:39.984 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:25:02.178 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-common-debuginfo-19.0.0-766.g0edf41a6.el9. 4.8 MB/s | 411 MB 01:26 2024-01-21T04:25:02.179 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:25:02.179 INFO:teuthology.orchestra.run.smithi144.stdout:Total 4.8 MB/s | 411 MB 01:26 2024-01-21T04:25:02.179 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:25:02.184 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:25:02.184 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:25:04.643 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:25:04.643 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:25:21.341 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:25:21.453 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:25:30.202 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:25:30.486 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:25:30.486 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:25:30.486 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:25:30.486 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:25:30.486 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:25:30.486 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:25:30.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-01-21T04:25:31.292 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:16 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:25:31.443 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:25:31.443 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-01-21T04:25:31.443 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:25:31.443 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-01-21T04:25:31.443 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:25:31.444 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.4 M 2024-01-21T04:25:31.444 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:25:31.444 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:25:31.444 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-01-21T04:25:31.444 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:25:31.444 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:25:31.444 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.4 M 2024-01-21T04:25:31.445 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 6.4 M 2024-01-21T04:25:31.445 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:25:31.765 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-76 4.5 MB/s | 1.4 MB 00:00 2024-01-21T04:25:31.766 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:25:31.766 INFO:teuthology.orchestra.run.smithi069.stdout:Total 4.4 MB/s | 1.4 MB 00:00 2024-01-21T04:25:31.766 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:25:31.770 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:25:31.770 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:25:31.812 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:25:31.812 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:25:32.064 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:25:32.163 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T04:25:32.555 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T04:25:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T04:25:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:25:32.825 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:25:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:25:32.825 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:25:32.825 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:25:33.025 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw-debuginfo 2024-01-21T04:25:33.585 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:18 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:25:33.735 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:25:33.735 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:25:33.735 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:25:33.735 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:25:33.735 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:25:33.735 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 170 M 2024-01-21T04:25:33.735 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:25:33.735 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:25:33.736 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:25:33.736 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:25:33.736 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:25:33.736 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 170 M 2024-01-21T04:25:33.736 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 721 M 2024-01-21T04:25:33.736 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:25:42.250 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-radosgw-debuginfo-19.0.0-766.g0edf41a6.el9 20 MB/s | 170 MB 00:08 2024-01-21T04:25:42.251 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:25:42.252 INFO:teuthology.orchestra.run.smithi069.stdout:Total 20 MB/s | 170 MB 00:08 2024-01-21T04:25:42.252 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:25:42.256 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:25:42.256 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:25:43.286 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:25:43.286 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:25:46.835 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:25:47.081 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:25:55.253 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:25:55.524 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:25:55.524 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:25:55.524 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:25:55.524 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:25:55.524 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:25:55.524 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:25:55.768 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-01-21T04:25:56.337 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:04:49 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:25:56.490 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:25:56.490 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.4 M 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout:======================================================================================= 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 1.4 M 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 6.4 M 2024-01-21T04:25:56.491 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:25:57.142 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-76 2.2 MB/s | 1.4 MB 00:00 2024-01-21T04:25:57.143 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:25:57.143 INFO:teuthology.orchestra.run.smithi144.stdout:Total 2.2 MB/s | 1.4 MB 00:00 2024-01-21T04:25:57.143 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:25:57.146 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:25:57.147 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:25:57.188 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:25:57.189 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:25:57.445 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:25:57.541 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T04:25:57.983 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T04:25:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g 1/1 2024-01-21T04:25:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:25:58.258 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:25:58.258 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:25:58.259 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:25:58.259 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:25:58.453 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-radosgw-debuginfo 2024-01-21T04:25:59.014 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:04:52 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:25:59.167 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:25:59.167 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:25:59.167 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:25:59.168 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:25:59.168 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:25:59.168 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 170 M 2024-01-21T04:25:59.168 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:25:59.168 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:25:59.168 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:25:59.168 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:25:59.168 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:25:59.168 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 170 M 2024-01-21T04:25:59.168 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 721 M 2024-01-21T04:25:59.168 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:26:00.697 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:26:00.996 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T04:26:05.444 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T04:26:05.718 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T04:26:05.718 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:26:05.718 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:26:05.718 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:26:05.718 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:26:05.718 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:26:05.921 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-test-debuginfo 2024-01-21T04:26:06.477 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:04:51 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:26:06.630 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:26:06.631 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:26:06.631 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:26:06.631 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:26:06.631 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:26:06.632 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 664 M 2024-01-21T04:26:06.632 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:26:06.632 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:26:06.632 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:26:06.632 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:26:06.632 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:26:06.632 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 664 M 2024-01-21T04:26:06.632 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 3.0 G 2024-01-21T04:26:06.632 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:26:45.438 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-test-debuginfo-19.0.0-766.g0edf41a6.el9.x8 17 MB/s | 664 MB 00:38 2024-01-21T04:26:45.439 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:26:45.439 INFO:teuthology.orchestra.run.smithi069.stdout:Total 17 MB/s | 664 MB 00:38 2024-01-21T04:26:45.440 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:26:45.445 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:26:45.445 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:26:45.697 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-radosgw-debuginfo-19.0.0-766.g0edf41a6.el9 3.7 MB/s | 170 MB 00:46 2024-01-21T04:26:45.698 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:26:45.698 INFO:teuthology.orchestra.run.smithi144.stdout:Total 3.7 MB/s | 170 MB 00:46 2024-01-21T04:26:45.699 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:26:45.702 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:26:45.703 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:26:46.734 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:26:46.735 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:26:49.343 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:26:49.343 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:27:04.151 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:27:04.334 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T04:27:08.807 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T04:27:09.109 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x8 1/1 2024-01-21T04:27:09.109 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:27:09.110 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:27:09.110 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:27:09.110 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:27:09.110 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:27:09.333 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-test-debuginfo 2024-01-21T04:27:09.895 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:06:02 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:27:10.048 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:27:10.048 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 664 M 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 664 M 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 3.0 G 2024-01-21T04:27:10.049 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:27:58.965 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:27:59.079 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:28:00.107 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-test-debuginfo-19.0.0-766.g0edf41a6.el9.x8 13 MB/s | 664 MB 00:50 2024-01-21T04:28:00.108 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:28:00.109 INFO:teuthology.orchestra.run.smithi144.stdout:Total 13 MB/s | 664 MB 00:50 2024-01-21T04:28:00.109 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:28:00.114 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:28:00.114 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:28:04.042 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:28:04.042 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:28:07.855 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:28:08.125 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:28:08.125 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:08.125 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:28:08.125 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:28:08.125 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:08.125 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:28:08.426 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-base-debuginfo 2024-01-21T04:28:08.993 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:53 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:28:09.094 INFO:teuthology.orchestra.run.smithi069.stdout:Package ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:28:09.145 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:28:09.146 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-01-21T04:28:09.146 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:28:09.228 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mgr-debuginfo 2024-01-21T04:28:09.780 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:06:54 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:28:09.933 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:28:09.934 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:09.934 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:28:09.934 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:09.934 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:28:09.934 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 20 M 2024-01-21T04:28:09.934 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:09.934 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:28:09.934 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:09.934 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:28:09.934 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:09.935 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 20 M 2024-01-21T04:28:09.935 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 102 M 2024-01-21T04:28:09.935 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:28:11.574 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mgr-debuginfo-19.0.0-766.g0edf41a6.el9.x86 12 MB/s | 20 MB 00:01 2024-01-21T04:28:11.575 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:28:11.576 INFO:teuthology.orchestra.run.smithi069.stdout:Total 12 MB/s | 20 MB 00:01 2024-01-21T04:28:11.576 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:28:11.579 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:28:11.579 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:28:11.726 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:28:11.726 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:28:14.008 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:28:14.164 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:28:15.126 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:28:15.389 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:28:15.389 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:15.389 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:28:15.389 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:28:15.389 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:15.389 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:28:15.557 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mds-debuginfo 2024-01-21T04:28:16.116 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:07:01 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:28:16.268 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:28:16.269 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:16.269 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:28:16.269 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:16.269 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:28:16.269 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 48 M 2024-01-21T04:28:16.269 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:16.269 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:28:16.269 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:16.269 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:28:16.269 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:16.269 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 48 M 2024-01-21T04:28:16.270 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 173 M 2024-01-21T04:28:16.270 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:28:20.852 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mds-debuginfo-19.0.0-766.g0edf41a6.el9.x86 10 MB/s | 48 MB 00:04 2024-01-21T04:28:20.852 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:28:20.853 INFO:teuthology.orchestra.run.smithi069.stdout:Total 10 MB/s | 48 MB 00:04 2024-01-21T04:28:20.853 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:28:20.857 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:28:20.857 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:28:21.171 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:28:21.171 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:28:25.896 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:28:26.251 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:28:27.621 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:28:27.906 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:28:27.906 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:27.906 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:28:27.906 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:28:27.906 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:27.906 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:28:28.085 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-mon-debuginfo 2024-01-21T04:28:28.647 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:07:13 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:28:28.798 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:28:28.799 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:28.799 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:28:28.799 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:28.799 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:28:28.799 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 99 M 2024-01-21T04:28:28.800 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:28.800 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:28:28.800 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:28.800 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:28:28.800 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:28.800 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 99 M 2024-01-21T04:28:28.800 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 449 M 2024-01-21T04:28:28.800 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:28:35.705 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-mon-debuginfo-19.0.0-766.g0edf41a6.el9.x86 14 MB/s | 99 MB 00:06 2024-01-21T04:28:35.706 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:28:35.706 INFO:teuthology.orchestra.run.smithi069.stdout:Total 14 MB/s | 99 MB 00:06 2024-01-21T04:28:35.706 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:28:35.710 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:28:35.711 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:28:36.316 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:28:36.316 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:28:46.537 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:28:46.627 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:28:49.511 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:28:49.785 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:28:49.785 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:49.785 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:28:49.785 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:28:49.785 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:49.785 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:28:49.979 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-osd-debuginfo 2024-01-21T04:28:50.541 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:07:35 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:28:50.691 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:28:50.692 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:50.692 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:28:50.692 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:50.692 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:28:50.692 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 286 M 2024-01-21T04:28:50.692 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:28:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:28:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:28:50.693 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:28:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 286 M 2024-01-21T04:28:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.2 G 2024-01-21T04:28:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:29:04.797 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-osd-debuginfo-19.0.0-766.g0edf41a6.el9.x86 20 MB/s | 286 MB 00:14 2024-01-21T04:29:04.798 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:29:04.798 INFO:teuthology.orchestra.run.smithi069.stdout:Total 20 MB/s | 286 MB 00:14 2024-01-21T04:29:04.798 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:29:04.802 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:29:04.802 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:29:06.509 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:29:06.510 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:29:13.822 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:29:13.922 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:29:23.840 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:29:24.104 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:29:24.104 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:24.104 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:29:24.104 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:29:24.104 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:24.104 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:29:24.372 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-base-debuginfo 2024-01-21T04:29:24.938 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:08:17 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:29:25.039 INFO:teuthology.orchestra.run.smithi144.stdout:Package ceph-base-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 is already installed. 2024-01-21T04:29:25.091 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:29:25.092 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2024-01-21T04:29:25.093 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:29:25.170 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mgr-debuginfo 2024-01-21T04:29:25.730 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:08:18 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:29:25.884 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:29:25.885 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:29:25.885 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:29:25.885 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:29:25.885 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:29:25.885 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 20 M 2024-01-21T04:29:25.885 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:25.886 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:29:25.886 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:29:25.886 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:29:25.886 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:25.886 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 20 M 2024-01-21T04:29:25.886 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 102 M 2024-01-21T04:29:25.886 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:29:27.518 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-mgr-debuginfo-19.0.0-766.g0edf41a6.el9.x86 12 MB/s | 20 MB 00:01 2024-01-21T04:29:27.518 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:29:27.519 INFO:teuthology.orchestra.run.smithi144.stdout:Total 12 MB/s | 20 MB 00:01 2024-01-21T04:29:27.519 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:29:27.523 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:29:27.523 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:29:27.670 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:29:27.671 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:29:29.936 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:29:30.040 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:29:31.052 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:29:31.317 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:29:31.318 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:31.318 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:29:31.318 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:29:31.318 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:31.318 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:29:31.487 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mds-debuginfo 2024-01-21T04:29:32.046 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:08:25 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:29:32.198 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:29:32.199 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:29:32.199 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:29:32.199 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:29:32.199 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:29:32.200 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 48 M 2024-01-21T04:29:32.200 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:32.200 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:29:32.200 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:29:32.200 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:29:32.200 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:32.200 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 48 M 2024-01-21T04:29:32.200 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 173 M 2024-01-21T04:29:32.200 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:29:34.762 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-mds-debuginfo-19.0.0-766.g0edf41a6.el9.x86 19 MB/s | 48 MB 00:02 2024-01-21T04:29:34.762 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:29:34.763 INFO:teuthology.orchestra.run.smithi144.stdout:Total 19 MB/s | 48 MB 00:02 2024-01-21T04:29:34.763 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:29:34.767 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:29:34.767 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:29:35.078 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:29:35.078 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:29:35.840 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:29:35.942 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:29:39.808 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:29:39.943 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:29:41.381 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:29:41.659 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:29:41.659 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:41.659 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:29:41.659 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:29:41.659 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:41.659 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:29:41.852 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-mon-debuginfo 2024-01-21T04:29:42.415 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:08:35 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:29:42.565 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:29:42.566 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:29:42.566 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:29:42.566 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:29:42.566 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:29:42.566 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 99 M 2024-01-21T04:29:42.566 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:42.566 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:29:42.567 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:29:42.567 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:29:42.567 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:29:42.567 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 99 M 2024-01-21T04:29:42.567 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 449 M 2024-01-21T04:29:42.567 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:29:43.338 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:29:43.612 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:29:43.612 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:43.612 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:29:43.612 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:29:43.612 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:43.612 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:29:43.833 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-fuse-debuginfo 2024-01-21T04:29:44.390 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:29 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:29:44.541 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:29:44.541 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.9 M 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 5.9 M 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 28 M 2024-01-21T04:29:44.542 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:29:46.304 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-fuse-debuginfo-19.0.0-766.g0edf41a6.el9.x8 3.4 MB/s | 5.9 MB 00:01 2024-01-21T04:29:46.305 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:29:46.305 INFO:teuthology.orchestra.run.smithi069.stdout:Total 3.4 MB/s | 5.9 MB 00:01 2024-01-21T04:29:46.306 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:29:46.309 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:29:46.309 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:29:46.380 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:29:46.380 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:29:47.115 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:29:47.227 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:29:47.572 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-mon-debuginfo-19.0.0-766.g0edf41a6.el9.x86 20 MB/s | 99 MB 00:05 2024-01-21T04:29:47.573 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:29:47.573 INFO:teuthology.orchestra.run.smithi144.stdout:Total 20 MB/s | 99 MB 00:05 2024-01-21T04:29:47.574 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:29:47.578 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:29:47.578 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:29:47.827 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:29:48.102 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:29:48.102 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:48.102 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:29:48.102 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:29:48.102 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:48.102 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:29:48.188 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:29:48.189 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:29:48.305 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados-devel-debuginfo 2024-01-21T04:29:48.863 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:33 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:29:49.013 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 109 k 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 109 k 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 439 k 2024-01-21T04:29:49.014 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:29:49.223 INFO:teuthology.orchestra.run.smithi069.stdout:librados-devel-debuginfo-19.0.0-766.g0edf41a6.e 521 kB/s | 109 kB 00:00 2024-01-21T04:29:49.224 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:29:49.224 INFO:teuthology.orchestra.run.smithi069.stdout:Total 518 kB/s | 109 kB 00:00 2024-01-21T04:29:49.224 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:29:49.227 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:29:49.227 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:29:49.260 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:29:49.260 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:29:49.378 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:29:49.480 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:29:49.843 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:29:50.115 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:29:50.115 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:50.116 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:29:50.116 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:29:50.116 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:50.116 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:29:50.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install libcephfs2-debuginfo 2024-01-21T04:29:50.863 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:35 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:29:51.014 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:29:51.015 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:51.016 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:29:51.016 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:51.016 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:29:51.016 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.3 M 2024-01-21T04:29:51.016 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:51.016 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:29:51.017 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:51.017 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:29:51.017 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:51.017 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 5.3 M 2024-01-21T04:29:51.017 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 25 M 2024-01-21T04:29:51.017 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:29:51.493 INFO:teuthology.orchestra.run.smithi069.stdout:libcephfs2-debuginfo-19.0.0-766.g0edf41a6.el9.x 11 MB/s | 5.3 MB 00:00 2024-01-21T04:29:51.494 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:29:51.494 INFO:teuthology.orchestra.run.smithi069.stdout:Total 11 MB/s | 5.3 MB 00:00 2024-01-21T04:29:51.494 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:29:51.498 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:29:51.498 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:29:51.562 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:29:51.562 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:29:52.233 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:29:52.324 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:29:52.797 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:29:53.086 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:29:53.086 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:53.086 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:29:53.086 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:29:53.086 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:53.086 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:29:53.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librados2-debuginfo 2024-01-21T04:29:53.834 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:38 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:29:53.985 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:29:53.986 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:53.986 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:29:53.986 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:53.986 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:29:53.986 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 44 M 2024-01-21T04:29:53.986 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:53.986 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:29:53.986 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:29:53.986 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:29:53.986 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:29:53.986 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 44 M 2024-01-21T04:29:53.987 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 252 M 2024-01-21T04:29:53.987 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:29:56.254 INFO:teuthology.orchestra.run.smithi069.stdout:librados2-debuginfo-19.0.0-766.g0edf41a6.el9.x8 19 MB/s | 44 MB 00:02 2024-01-21T04:29:56.255 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:29:56.255 INFO:teuthology.orchestra.run.smithi069.stdout:Total 19 MB/s | 44 MB 00:02 2024-01-21T04:29:56.256 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:29:56.259 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:29:56.260 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:29:56.561 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:29:56.561 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:29:58.478 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:29:58.586 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:01.566 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:01.583 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:30:01.677 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:30:01.844 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:01.844 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:30:01.844 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:30:01.844 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:30:01.844 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:30:01.844 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:30:02.060 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-osd-debuginfo 2024-01-21T04:30:02.625 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:08:55 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:30:02.778 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:30:02.778 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 286 M 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 286 M 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 1.2 G 2024-01-21T04:30:02.779 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:30:03.488 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:30:03.767 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:30:03.767 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:03.767 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:30:03.767 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:30:03.767 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:03.767 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:30:03.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install librbd1-debuginfo 2024-01-21T04:30:04.561 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:08:49 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:30:04.711 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:30:04.711 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:04.711 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:30:04.712 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:04.712 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:30:04.712 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 52 M 2024-01-21T04:30:04.712 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:04.712 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:30:04.712 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:04.712 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:30:04.712 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:04.712 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 52 M 2024-01-21T04:30:04.712 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 308 M 2024-01-21T04:30:04.712 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:30:12.039 INFO:teuthology.orchestra.run.smithi069.stdout:librbd1-debuginfo-19.0.0-766.g0edf41a6.el9.x86_ 7.1 MB/s | 52 MB 00:07 2024-01-21T04:30:12.040 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:30:12.040 INFO:teuthology.orchestra.run.smithi069.stdout:Total 7.1 MB/s | 52 MB 00:07 2024-01-21T04:30:12.041 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:30:12.045 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:30:12.045 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:30:12.393 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:30:12.393 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:30:17.088 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-osd-debuginfo-19.0.0-766.g0edf41a6.el9.x86 20 MB/s | 286 MB 00:14 2024-01-21T04:30:17.089 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:30:17.090 INFO:teuthology.orchestra.run.smithi144.stdout:Total 20 MB/s | 286 MB 00:14 2024-01-21T04:30:17.090 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:30:17.094 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:30:17.094 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:30:18.375 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:30:18.504 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:18.803 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:30:18.803 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:30:20.660 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:20.945 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:20.945 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:20.945 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:30:20.945 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:30:20.945 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:20.945 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:30:21.134 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-cephfs-debuginfo 2024-01-21T04:30:21.688 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:09:06 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:30:21.841 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:30:21.841 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:21.841 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:30:21.841 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:21.841 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:30:21.841 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 615 k 2024-01-21T04:30:21.841 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:21.842 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:30:21.842 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:21.842 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:30:21.842 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:21.842 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 615 k 2024-01-21T04:30:21.842 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.1 M 2024-01-21T04:30:21.842 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:30:22.125 INFO:teuthology.orchestra.run.smithi069.stdout:python3-cephfs-debuginfo-19.0.0-766.g0edf41a6.e 2.1 MB/s | 615 kB 00:00 2024-01-21T04:30:22.125 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:30:22.126 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.1 MB/s | 615 kB 00:00 2024-01-21T04:30:22.126 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:30:22.129 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:30:22.129 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:30:22.166 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:30:22.166 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:30:22.330 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:30:22.449 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:30:22.833 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:30:23.103 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:30:23.103 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:23.103 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:30:23.103 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:30:23.103 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:23.103 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:30:23.300 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rados-debuginfo 2024-01-21T04:30:23.858 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:09:08 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:30:24.008 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 769 k 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 769 k 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.7 M 2024-01-21T04:30:24.009 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:30:24.436 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rados-debuginfo-19.0.0-766.g0edf41a6.el 1.8 MB/s | 769 kB 00:00 2024-01-21T04:30:24.437 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:30:24.437 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.8 MB/s | 769 kB 00:00 2024-01-21T04:30:24.437 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:30:24.441 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:30:24.441 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:30:24.478 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:30:24.478 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:30:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:30:24.759 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T04:30:25.147 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T04:30:25.429 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T04:30:25.429 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:25.429 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:30:25.429 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:30:25.429 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:25.429 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:30:25.624 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rbd-debuginfo 2024-01-21T04:30:26.183 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:09:11 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:30:26.335 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:30:26.336 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:26.336 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:30:26.336 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:26.336 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:30:26.336 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.1 M 2024-01-21T04:30:26.336 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:26.336 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:30:26.336 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:26.337 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:30:26.337 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:26.337 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.1 M 2024-01-21T04:30:26.337 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 4.3 M 2024-01-21T04:30:26.337 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:30:26.631 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rbd-debuginfo-19.0.0-766.g0edf41a6.el9. 3.8 MB/s | 1.1 MB 00:00 2024-01-21T04:30:26.632 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:30:26.632 INFO:teuthology.orchestra.run.smithi069.stdout:Total 3.8 MB/s | 1.1 MB 00:00 2024-01-21T04:30:26.632 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:30:26.635 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:30:26.636 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:30:26.674 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:30:26.674 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:30:26.892 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:30:26.996 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:30:27.407 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:30:27.683 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:30:27.683 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:27.683 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:30:27.683 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:30:27.683 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:27.683 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:30:27.888 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install python3-rgw-debuginfo 2024-01-21T04:30:28.448 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:09:13 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:30:28.600 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:30:28.600 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:28.600 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:30:28.601 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:28.601 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:30:28.601 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 337 k 2024-01-21T04:30:28.601 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:28.601 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:30:28.601 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:28.601 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:30:28.601 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:28.601 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 337 k 2024-01-21T04:30:28.601 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 1.0 M 2024-01-21T04:30:28.601 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:30:28.993 INFO:teuthology.orchestra.run.smithi069.stdout:python3-rgw-debuginfo-19.0.0-766.g0edf41a6.el9. 862 kB/s | 337 kB 00:00 2024-01-21T04:30:28.993 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:30:28.993 INFO:teuthology.orchestra.run.smithi069.stdout:Total 859 kB/s | 337 kB 00:00 2024-01-21T04:30:28.993 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:30:28.997 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:30:28.997 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:30:29.030 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:30:29.031 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:30:29.169 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:30:29.275 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:30:29.663 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:30:29.935 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:30:29.935 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:29.936 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:30:29.936 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:30:29.936 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:29.936 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:30:30.161 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-fuse-debuginfo 2024-01-21T04:30:30.716 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:09:15 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:30:30.867 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:30:30.868 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:30.868 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:30:30.868 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:30.868 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:30:30.868 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 509 k 2024-01-21T04:30:30.868 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:30.868 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:30:30.868 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:30.868 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:30:30.868 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:30.868 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 509 k 2024-01-21T04:30:30.869 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 2.5 M 2024-01-21T04:30:30.869 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:30:31.254 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-fuse-debuginfo-19.0.0-766.g0edf41a6.el9.x86 1.3 MB/s | 509 kB 00:00 2024-01-21T04:30:31.255 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:30:31.255 INFO:teuthology.orchestra.run.smithi069.stdout:Total 1.3 MB/s | 509 kB 00:00 2024-01-21T04:30:31.255 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:30:31.258 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:30:31.258 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:30:31.294 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:30:31.295 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:30:31.448 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:30:31.545 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:31.932 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:32.209 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:32.209 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:32.209 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:30:32.209 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:30:32.210 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:32.210 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:30:32.410 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-mirror-debuginfo 2024-01-21T04:30:32.972 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:09:17 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:30:33.124 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:30:33.125 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:33.125 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:30:33.125 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:33.125 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:30:33.125 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 52 M 2024-01-21T04:30:33.125 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:33.125 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:30:33.125 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:33.125 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:30:33.126 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:33.126 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 52 M 2024-01-21T04:30:33.126 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 298 M 2024-01-21T04:30:33.126 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:30:35.682 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-mirror-debuginfo-19.0.0-766.g0edf41a6.el9.x 20 MB/s | 52 MB 00:02 2024-01-21T04:30:35.683 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:30:35.683 INFO:teuthology.orchestra.run.smithi069.stdout:Total 20 MB/s | 52 MB 00:02 2024-01-21T04:30:35.684 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:30:35.687 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:30:35.688 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:30:36.022 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:30:36.022 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:30:41.897 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:30:42.205 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:30:44.250 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:30:44.533 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:30:44.533 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:44.533 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:30:44.533 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:30:44.533 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:44.533 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:30:44.726 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install rbd-nbd-debuginfo 2024-01-21T04:30:45.293 INFO:teuthology.orchestra.run.smithi069.stdout:Last metadata expiration check: 0:09:30 ago on Sun 21 Jan 2024 04:21:15 AM UTC. 2024-01-21T04:30:45.443 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-01-21T04:30:45.444 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:45.444 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repo Size 2024-01-21T04:30:45.444 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:45.444 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-01-21T04:30:45.444 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.3 M 2024-01-21T04:30:45.444 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:45.444 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-01-21T04:30:45.444 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-01-21T04:30:45.444 INFO:teuthology.orchestra.run.smithi069.stdout:Install 1 Package 2024-01-21T04:30:45.444 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:45.444 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 1.3 M 2024-01-21T04:30:45.445 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 5.5 M 2024-01-21T04:30:45.445 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-01-21T04:30:46.067 INFO:teuthology.orchestra.run.smithi069.stdout:rbd-nbd-debuginfo-19.0.0-766.g0edf41a6.el9.x86_ 2.0 MB/s | 1.3 MB 00:00 2024-01-21T04:30:46.068 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:30:46.068 INFO:teuthology.orchestra.run.smithi069.stdout:Total 2.0 MB/s | 1.3 MB 00:00 2024-01-21T04:30:46.069 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-01-21T04:30:46.072 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-01-21T04:30:46.072 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-01-21T04:30:46.114 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-01-21T04:30:46.114 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-01-21T04:30:46.368 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-01-21T04:30:46.466 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:46.936 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:47.216 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:47.217 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:47.217 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-01-21T04:30:47.217 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:30:47.217 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:30:47.217 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-01-21T04:30:47.399 DEBUG:teuthology.parallel:result is None 2024-01-21T04:30:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:30:48.450 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:55.898 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:30:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:30:56.184 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:30:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:30:56.184 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:30:56.184 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:30:56.432 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-fuse-debuginfo 2024-01-21T04:30:56.995 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:09:49 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:30:57.146 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:30:57.147 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:30:57.147 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:30:57.148 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:30:57.148 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:30:57.148 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.9 M 2024-01-21T04:30:57.148 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:30:57.148 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:30:57.148 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:30:57.148 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:30:57.148 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:30:57.148 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 5.9 M 2024-01-21T04:30:57.148 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 28 M 2024-01-21T04:30:57.148 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:30:58.303 INFO:teuthology.orchestra.run.smithi144.stdout:ceph-fuse-debuginfo-19.0.0-766.g0edf41a6.el9.x8 5.1 MB/s | 5.9 MB 00:01 2024-01-21T04:30:58.304 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:30:58.304 INFO:teuthology.orchestra.run.smithi144.stdout:Total 5.1 MB/s | 5.9 MB 00:01 2024-01-21T04:30:58.305 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:30:58.308 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:30:58.308 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:30:58.376 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:30:58.376 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:30:59.107 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:30:59.201 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:30:59.797 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:31:00.076 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:31:00.076 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:00.076 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:00.076 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:00.076 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:00.076 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:00.274 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librados-devel-debuginfo 2024-01-21T04:31:00.837 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:09:53 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:31:00.988 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:31:00.988 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 109 k 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 109 k 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 439 k 2024-01-21T04:31:00.989 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:31:01.314 INFO:teuthology.orchestra.run.smithi144.stdout:librados-devel-debuginfo-19.0.0-766.g0edf41a6.e 334 kB/s | 109 kB 00:00 2024-01-21T04:31:01.315 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:31:01.315 INFO:teuthology.orchestra.run.smithi144.stdout:Total 333 kB/s | 109 kB 00:00 2024-01-21T04:31:01.315 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:31:01.318 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:31:01.318 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:31:01.353 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:31:01.353 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:31:01.469 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:31:01.571 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:31:01.975 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:31:02.245 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:31:02.245 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:02.245 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:02.245 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:02.245 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:02.245 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:02.456 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install libcephfs2-debuginfo 2024-01-21T04:31:03.017 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:09:56 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:31:03.170 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:31:03.171 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:03.171 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:31:03.171 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:03.171 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:31:03.171 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 5.3 M 2024-01-21T04:31:03.171 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:03.171 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:31:03.171 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:03.172 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:31:03.172 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:03.172 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 5.3 M 2024-01-21T04:31:03.172 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 25 M 2024-01-21T04:31:03.172 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:31:03.792 INFO:teuthology.orchestra.run.smithi144.stdout:libcephfs2-debuginfo-19.0.0-766.g0edf41a6.el9.x 8.6 MB/s | 5.3 MB 00:00 2024-01-21T04:31:03.793 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:31:03.794 INFO:teuthology.orchestra.run.smithi144.stdout:Total 8.6 MB/s | 5.3 MB 00:00 2024-01-21T04:31:03.794 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:31:03.797 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:31:03.797 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:31:03.860 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:31:03.861 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:31:04.532 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:31:04.666 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:31:05.209 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:31:05.518 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:31:05.518 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:05.518 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:05.518 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:05.518 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:05.518 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:05.717 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librados2-debuginfo 2024-01-21T04:31:06.277 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:09:59 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:31:06.429 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:31:06.430 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:06.430 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:31:06.430 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:06.431 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:31:06.431 INFO:teuthology.orchestra.run.smithi144.stdout: librados2-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 44 M 2024-01-21T04:31:06.431 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:06.431 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:31:06.431 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:06.431 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:31:06.431 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:06.431 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 44 M 2024-01-21T04:31:06.431 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 252 M 2024-01-21T04:31:06.431 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:31:08.745 INFO:teuthology.orchestra.run.smithi144.stdout:librados2-debuginfo-19.0.0-766.g0edf41a6.el9.x8 19 MB/s | 44 MB 00:02 2024-01-21T04:31:08.745 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:31:08.746 INFO:teuthology.orchestra.run.smithi144.stdout:Total 19 MB/s | 44 MB 00:02 2024-01-21T04:31:08.746 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:31:08.750 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:31:08.750 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:31:09.054 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:31:09.054 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:31:14.089 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:31:14.184 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:31:16.090 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:31:16.365 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_6 1/1 2024-01-21T04:31:16.365 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:16.365 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:16.365 INFO:teuthology.orchestra.run.smithi144.stdout: librados2-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:16.366 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:16.366 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:16.574 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install librbd1-debuginfo 2024-01-21T04:31:17.142 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:10:10 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:31:17.294 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:31:17.294 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:17.294 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:31:17.294 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:17.295 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:31:17.295 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 52 M 2024-01-21T04:31:17.295 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:17.295 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:31:17.295 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:17.295 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:31:17.295 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:17.295 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 52 M 2024-01-21T04:31:17.295 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 308 M 2024-01-21T04:31:17.295 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:31:20.223 INFO:teuthology.orchestra.run.smithi144.stdout:librbd1-debuginfo-19.0.0-766.g0edf41a6.el9.x86_ 18 MB/s | 52 MB 00:02 2024-01-21T04:31:20.223 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:31:20.224 INFO:teuthology.orchestra.run.smithi144.stdout:Total 18 MB/s | 52 MB 00:02 2024-01-21T04:31:20.224 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:31:20.228 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:31:20.228 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:31:20.574 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:31:20.574 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:31:26.583 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:31:26.680 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:31:28.914 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:31:29.206 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:31:29.206 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:29.207 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:29.207 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:29.207 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:29.207 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:29.413 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-cephfs-debuginfo 2024-01-21T04:31:29.985 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:10:22 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:31:30.146 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:31:30.147 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:30.147 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:31:30.147 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:30.147 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:31:30.147 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 615 k 2024-01-21T04:31:30.147 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:30.147 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:31:30.147 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:30.147 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:31:30.148 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:30.148 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 615 k 2024-01-21T04:31:30.148 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 2.1 M 2024-01-21T04:31:30.148 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:31:30.425 INFO:teuthology.orchestra.run.smithi144.stdout:python3-cephfs-debuginfo-19.0.0-766.g0edf41a6.e 2.2 MB/s | 615 kB 00:00 2024-01-21T04:31:30.425 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:31:30.426 INFO:teuthology.orchestra.run.smithi144.stdout:Total 2.2 MB/s | 615 kB 00:00 2024-01-21T04:31:30.426 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:31:30.429 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:31:30.430 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:31:30.465 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:31:30.466 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:31:30.633 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:31:30.740 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:31:31.154 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:31:31.425 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9. 1/1 2024-01-21T04:31:31.425 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:31.425 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:31.425 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:31.425 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:31.425 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:31.614 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rados-debuginfo 2024-01-21T04:31:32.176 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:10:25 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:31:32.330 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:31:32.330 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:32.330 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:31:32.330 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:32.331 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:31:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 769 k 2024-01-21T04:31:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:32.331 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:31:32.331 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:32.331 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:31:32.331 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:32.332 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 769 k 2024-01-21T04:31:32.332 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 2.7 M 2024-01-21T04:31:32.332 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:31:32.612 INFO:teuthology.orchestra.run.smithi144.stdout:python3-rados-debuginfo-19.0.0-766.g0edf41a6.el 2.7 MB/s | 769 kB 00:00 2024-01-21T04:31:32.613 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:31:32.613 INFO:teuthology.orchestra.run.smithi144.stdout:Total 2.7 MB/s | 769 kB 00:00 2024-01-21T04:31:32.613 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:31:32.616 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:31:32.616 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:31:32.654 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:31:32.654 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:31:32.837 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:31:32.935 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T04:31:33.318 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T04:31:33.592 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x 1/1 2024-01-21T04:31:33.592 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:33.592 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:33.592 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:33.592 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:33.593 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:33.795 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rbd-debuginfo 2024-01-21T04:31:34.356 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:10:27 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:31:34.507 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:31:34.508 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:34.508 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:31:34.508 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:34.508 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:31:34.508 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.1 M 2024-01-21T04:31:34.508 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:34.508 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:31:34.508 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:34.508 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:31:34.509 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:34.509 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 1.1 M 2024-01-21T04:31:34.509 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 4.3 M 2024-01-21T04:31:34.509 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:31:34.807 INFO:teuthology.orchestra.run.smithi144.stdout:python3-rbd-debuginfo-19.0.0-766.g0edf41a6.el9. 3.8 MB/s | 1.1 MB 00:00 2024-01-21T04:31:34.808 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:31:34.808 INFO:teuthology.orchestra.run.smithi144.stdout:Total 3.8 MB/s | 1.1 MB 00:00 2024-01-21T04:31:34.808 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:31:34.811 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:31:34.811 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:31:34.851 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:31:34.851 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:31:35.065 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:31:35.172 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:31:35.587 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:31:35.886 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:31:35.886 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:35.886 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:35.886 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:35.886 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:35.886 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:36.115 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install python3-rgw-debuginfo 2024-01-21T04:31:36.678 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:10:29 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:31:36.831 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 337 k 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 337 k 2024-01-21T04:31:36.832 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 1.0 M 2024-01-21T04:31:36.833 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:31:37.377 INFO:teuthology.orchestra.run.smithi144.stdout:python3-rgw-debuginfo-19.0.0-766.g0edf41a6.el9. 619 kB/s | 337 kB 00:00 2024-01-21T04:31:37.377 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:31:37.377 INFO:teuthology.orchestra.run.smithi144.stdout:Total 618 kB/s | 337 kB 00:00 2024-01-21T04:31:37.378 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:31:37.381 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:31:37.381 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:31:37.415 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:31:37.416 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:31:37.560 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:31:37.658 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:31:38.051 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:31:38.362 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86 1/1 2024-01-21T04:31:38.362 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:38.362 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:38.362 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:38.362 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:38.362 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:38.574 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-fuse-debuginfo 2024-01-21T04:31:39.141 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:10:32 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:31:39.293 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:31:39.294 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:39.294 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:31:39.295 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:39.295 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:31:39.295 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 509 k 2024-01-21T04:31:39.295 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:39.295 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:31:39.295 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:39.295 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:31:39.295 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:39.295 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 509 k 2024-01-21T04:31:39.295 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 2.5 M 2024-01-21T04:31:39.295 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:31:39.963 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-fuse-debuginfo-19.0.0-766.g0edf41a6.el9.x86 761 kB/s | 509 kB 00:00 2024-01-21T04:31:39.964 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:31:39.964 INFO:teuthology.orchestra.run.smithi144.stdout:Total 760 kB/s | 509 kB 00:00 2024-01-21T04:31:39.964 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:31:39.968 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:31:39.968 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:31:40.004 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:31:40.004 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:31:40.166 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:31:40.270 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:31:40.718 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:31:41.000 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:31:41.001 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:41.001 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:41.001 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:41.001 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:41.001 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:41.185 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-mirror-debuginfo 2024-01-21T04:31:41.752 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:10:34 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:31:41.903 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:31:41.904 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:41.904 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:31:41.904 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:41.904 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:31:41.904 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 52 M 2024-01-21T04:31:41.905 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:41.905 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:31:41.905 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:41.905 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:31:41.905 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:41.905 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 52 M 2024-01-21T04:31:41.905 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 298 M 2024-01-21T04:31:41.905 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:31:44.683 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-mirror-debuginfo-19.0.0-766.g0edf41a6.el9.x 19 MB/s | 52 MB 00:02 2024-01-21T04:31:44.684 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:31:44.684 INFO:teuthology.orchestra.run.smithi144.stdout:Total 19 MB/s | 52 MB 00:02 2024-01-21T04:31:44.684 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:31:44.688 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:31:44.689 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:31:45.024 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:31:45.025 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:31:50.926 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:31:51.022 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:31:53.168 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:31:53.434 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_ 1/1 2024-01-21T04:31:53.435 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:53.435 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:53.435 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:53.435 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:53.435 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:53.636 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install rbd-nbd-debuginfo 2024-01-21T04:31:54.205 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:10:47 ago on Sun 21 Jan 2024 04:21:07 AM UTC. 2024-01-21T04:31:54.357 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-01-21T04:31:54.357 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repo Size 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-766.g0edf41a6.el9 ceph 1.3 M 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout:Install 1 Package 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 1.3 M 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 5.5 M 2024-01-21T04:31:54.358 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-01-21T04:31:54.685 INFO:teuthology.orchestra.run.smithi144.stdout:rbd-nbd-debuginfo-19.0.0-766.g0edf41a6.el9.x86_ 3.8 MB/s | 1.3 MB 00:00 2024-01-21T04:31:54.685 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-01-21T04:31:54.686 INFO:teuthology.orchestra.run.smithi144.stdout:Total 3.8 MB/s | 1.3 MB 00:00 2024-01-21T04:31:54.686 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-01-21T04:31:54.689 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-01-21T04:31:54.689 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-01-21T04:31:54.733 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-01-21T04:31:54.733 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-01-21T04:31:54.968 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-01-21T04:31:55.067 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:31:55.537 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:31:55.823 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 1/1 2024-01-21T04:31:55.823 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:55.823 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-01-21T04:31:55.823 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd-debuginfo-2:19.0.0-766.g0edf41a6.el9.x86_64 2024-01-21T04:31:55.823 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:55.823 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-01-21T04:31:56.004 DEBUG:teuthology.parallel:result is None 2024-01-21T04:31:56.005 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T04:31:56.248 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T04:31:56.273 INFO:teuthology.orchestra.run.smithi069.stdout:19.0.0-766.g0edf41a6.el9 2024-01-21T04:31:56.274 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-766.g0edf41a6.el9 2024-01-21T04:31:56.274 INFO:teuthology.task.install:The correct ceph version 19.0.0-766.g0edf41a6 is installed. 2024-01-21T04:31:56.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T04:31:56.419 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-21T04:31:56.440 INFO:teuthology.orchestra.run.smithi144.stdout:19.0.0-766.g0edf41a6.el9 2024-01-21T04:31:56.440 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-766.g0edf41a6.el9 2024-01-21T04:31:56.440 INFO:teuthology.task.install:The correct ceph version 19.0.0-766.g0edf41a6 is installed. 2024-01-21T04:31:56.442 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-21T04:31:56.442 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:31:56.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T04:31:56.474 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:31:56.474 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-21T04:31:56.508 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-21T04:31:56.508 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:31:56.508 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T04:31:56.547 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T04:31:56.615 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:31:56.615 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/daemon-helper 2024-01-21T04:31:56.643 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-21T04:31:56.709 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-21T04:31:56.709 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:31:56.709 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T04:31:56.738 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T04:31:56.806 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:31:56.806 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-21T04:31:56.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-21T04:31:56.899 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-21T04:31:56.899 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:31:56.899 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T04:31:56.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T04:31:56.993 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:31:56.993 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/stdin-killer 2024-01-21T04:31:57.020 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-21T04:31:57.086 INFO:teuthology.run_tasks:Running task ceph... 2024-01-21T04:31:57.166 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-21T04:31:57.166 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2024-01-21T04:31:57.169 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 777 /var/log/ceph 2024-01-21T04:31:57.195 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-21T04:31:57.195 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-21T04:31:57.237 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-21T04:31:57.263 INFO:tasks.ceph:Creating extra log directories... 2024-01-21T04:31:57.263 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-21T04:31:57.304 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-21T04:31:57.341 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-21T04:31:57.341 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-21T04:31:57.341 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711', 'branch': 'wip-yuri8-testing-2024-01-18-0823', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7523711', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri8-testing-2024-01-18-0823'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri8-testing-2024-01-18-0823', 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9599, 'sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri8-testing-2024-01-18-0823', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '0edf41a622739843d6b978b179ff3227b476dd9d', 'targets': {'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJHhHBSinmQEofBQijDsym1K+SkxvxwWk9w04OcLVikHqZ6Uhn4Og3fI/aWrXrLEWArQdCj/i62a0r94q3wei3Q=', 'smithi144.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPS4pNmIUfUvxeA5qS72DYwNe/zOtn6/GnN+cuB4nt/rM9D0AzxuESufov7R1JEN2b2pqSV503uetyIVzM9KDoA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-19_16:35:08', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-21T04:31:57.342 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-21T04:31:57.371 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-21T04:31:57.386 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2024-01-21T04:31:57.426 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m0777 -- /var/run/ceph 2024-01-21T04:31:57.453 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:31:57.453 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-01-21T04:31:57.505 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-21T04:31:57.505 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-01-21T04:31:57.560 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-21T04:31:57.560 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T04:31:57.560 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-01-21T04:31:57.560 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T04:31:57.560 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T04:31:57.560 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-01-21 04:23:02.545520955 +0000 2024-01-21T04:31:57.560 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-01-21 04:20:20.981700436 +0000 2024-01-21T04:31:57.560 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-01-21 04:20:20.981700436 +0000 2024-01-21T04:31:57.560 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-01-21 04:20:20.981700436 +0000 2024-01-21T04:31:57.560 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-21T04:31:57.624 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-01-21T04:31:57.624 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-01-21T04:31:57.624 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000202386 s, 2.5 MB/s 2024-01-21T04:31:57.625 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-21T04:31:57.680 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-01-21T04:31:57.735 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-21T04:31:57.735 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T04:31:57.735 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-01-21T04:31:57.735 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T04:31:57.735 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T04:31:57.735 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-01-21 04:23:02.546520932 +0000 2024-01-21T04:31:57.735 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-01-21 04:20:21.332692211 +0000 2024-01-21T04:31:57.735 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-01-21 04:20:21.333692187 +0000 2024-01-21T04:31:57.735 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-01-21 04:20:21.332692211 +0000 2024-01-21T04:31:57.736 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-21T04:31:57.800 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-01-21T04:31:57.800 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-01-21T04:31:57.800 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000184582 s, 2.8 MB/s 2024-01-21T04:31:57.801 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-21T04:31:57.856 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-01-21T04:31:57.910 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-21T04:31:57.911 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T04:31:57.911 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-01-21T04:31:57.911 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T04:31:57.911 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T04:31:57.911 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-01-21 04:23:02.546520932 +0000 2024-01-21T04:31:57.911 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-01-21 04:20:21.667684360 +0000 2024-01-21T04:31:57.911 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-01-21 04:20:21.667684360 +0000 2024-01-21T04:31:57.911 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-01-21 04:20:21.667684360 +0000 2024-01-21T04:31:57.911 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-21T04:31:57.976 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-01-21T04:31:57.977 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-01-21T04:31:57.977 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000202958 s, 2.5 MB/s 2024-01-21T04:31:57.978 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-21T04:31:58.033 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-01-21T04:31:58.088 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-21T04:31:58.088 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T04:31:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-01-21T04:31:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T04:31:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T04:31:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-01-21 04:23:02.547520910 +0000 2024-01-21T04:31:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-01-21 04:20:21.991676768 +0000 2024-01-21T04:31:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-01-21 04:20:21.991676768 +0000 2024-01-21T04:31:58.088 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-01-21 04:20:21.991676768 +0000 2024-01-21T04:31:58.089 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-21T04:31:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-01-21T04:31:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-01-21T04:31:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000207282 s, 2.5 MB/s 2024-01-21T04:31:58.154 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-21T04:31:58.209 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-21T04:31:58.209 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:31:58.209 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/scratch_devs of=/dev/stdout 2024-01-21T04:31:58.225 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-21T04:31:58.226 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_1 2024-01-21T04:31:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-21T04:31:58.281 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T04:31:58.282 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-01-21T04:31:58.282 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T04:31:58.282 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T04:31:58.282 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-01-21 04:23:07.576612646 +0000 2024-01-21T04:31:58.282 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-01-21 04:20:20.982475451 +0000 2024-01-21T04:31:58.282 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-01-21 04:20:20.982475451 +0000 2024-01-21T04:31:58.282 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-01-21 04:20:20.982475451 +0000 2024-01-21T04:31:58.282 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-21T04:31:58.349 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-01-21T04:31:58.349 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-01-21T04:31:58.349 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000214126 s, 2.4 MB/s 2024-01-21T04:31:58.350 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-21T04:31:58.407 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_2 2024-01-21T04:31:58.461 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-21T04:31:58.462 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T04:31:58.462 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-01-21T04:31:58.462 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T04:31:58.462 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T04:31:58.462 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-01-21 04:23:07.576612646 +0000 2024-01-21T04:31:58.462 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-01-21 04:20:21.332469371 +0000 2024-01-21T04:31:58.462 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-01-21 04:20:21.332469371 +0000 2024-01-21T04:31:58.462 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-01-21 04:20:21.332469371 +0000 2024-01-21T04:31:58.462 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-21T04:31:58.527 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-01-21T04:31:58.528 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-01-21T04:31:58.528 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000160775 s, 3.2 MB/s 2024-01-21T04:31:58.529 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-21T04:31:58.584 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_3 2024-01-21T04:31:58.639 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-21T04:31:58.639 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T04:31:58.639 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-01-21T04:31:58.639 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T04:31:58.639 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T04:31:58.640 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-01-21 04:23:07.577612629 +0000 2024-01-21T04:31:58.640 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-01-21 04:20:21.671463482 +0000 2024-01-21T04:31:58.640 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-01-21 04:20:21.671463482 +0000 2024-01-21T04:31:58.640 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-01-21 04:20:21.671463482 +0000 2024-01-21T04:31:58.640 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-21T04:31:58.704 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-01-21T04:31:58.705 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-01-21T04:31:58.705 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000182338 s, 2.8 MB/s 2024-01-21T04:31:58.706 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-21T04:31:58.761 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_4 2024-01-21T04:31:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-21T04:31:58.815 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-21T04:31:58.815 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-01-21T04:31:58.816 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-21T04:31:58.816 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-01-21T04:31:58.816 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-01-21 04:23:07.577612629 +0000 2024-01-21T04:31:58.816 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-01-21 04:20:22.011457576 +0000 2024-01-21T04:31:58.816 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-01-21 04:20:22.011457576 +0000 2024-01-21T04:31:58.816 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-01-21 04:20:22.011457576 +0000 2024-01-21T04:31:58.816 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-21T04:31:58.881 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-01-21T04:31:58.881 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-01-21T04:31:58.881 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000182265 s, 2.8 MB/s 2024-01-21T04:31:58.882 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-21T04:31:58.938 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-21T04:31:58.938 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-21T04:31:58.938 INFO:tasks.ceph:Generating config... 2024-01-21T04:31:58.939 INFO:tasks.ceph:[client] debug monc = 20 2024-01-21T04:31:58.940 INFO:tasks.ceph:[client] debug ms = 1 2024-01-21T04:31:58.940 INFO:tasks.ceph:[client] debug objecter = 20 2024-01-21T04:31:58.940 INFO:tasks.ceph:[client] debug rados = 20 2024-01-21T04:31:58.940 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-01-21T04:31:58.940 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-01-21T04:31:58.940 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-01-21T04:31:58.940 INFO:tasks.ceph:[global] ms type = async 2024-01-21T04:31:58.940 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-01-21T04:31:58.940 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-21T04:31:58.940 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-21T04:31:58.940 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-21T04:31:58.940 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-21T04:31:58.941 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-21T04:31:58.941 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-01-21T04:31:58.941 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-01-21T04:31:58.941 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-01-21T04:31:58.941 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-01-21T04:31:58.941 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-01-21T04:31:58.941 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-01-21T04:31:58.941 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-01-21T04:31:58.941 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-01-21T04:31:58.941 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-21T04:31:58.941 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-21T04:31:58.941 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-21T04:31:58.941 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-21T04:31:58.941 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] debug monc = 20 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] osd class default list = * 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] osd class load list = * 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-01-21T04:31:58.942 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-01-21T04:31:58.943 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-21T04:31:58.943 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-01-21T04:31:58.943 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-01-21T04:31:58.943 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-01-21T04:31:58.943 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-01-21T04:31:58.943 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-21T04:31:58.943 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-01-21T04:31:58.943 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-01-21T04:31:58.943 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-21T04:31:58.943 INFO:tasks.ceph:Setting up mon.a... 2024-01-21T04:31:58.943 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-21T04:31:59.001 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.keyring 2024-01-21T04:31:59.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-21T04:31:59.050 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-21T04:31:59.077 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.69'), ('mon.c', '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]'), ('mon.b', '172.21.15.144')] 2024-01-21T04:31:59.078 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790],172.21.15.144', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-01-21T04:31:59.079 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:31:59.079 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-21T04:31:59.132 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.69 --addv c '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]' --add b 172.21.15.144 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:31:59.217 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:31:59.217 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: generated fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:31:59.217 INFO:teuthology.orchestra.run.smithi069.stdout:setting min_mon_release = quincy 2024-01-21T04:31:59.217 INFO:teuthology.orchestra.run.smithi069.stdout:epoch 0 2024-01-21T04:31:59.217 INFO:teuthology.orchestra.run.smithi069.stdout:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:31:59.217 INFO:teuthology.orchestra.run.smithi069.stdout:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:31:59.217 INFO:teuthology.orchestra.run.smithi069.stdout:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:31:59.218 INFO:teuthology.orchestra.run.smithi069.stdout:min_mon_release 17 (quincy) 2024-01-21T04:31:59.218 INFO:teuthology.orchestra.run.smithi069.stdout:election_strategy: 1 2024-01-21T04:31:59.218 INFO:teuthology.orchestra.run.smithi069.stdout:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:31:59.218 INFO:teuthology.orchestra.run.smithi069.stdout:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:31:59.218 INFO:teuthology.orchestra.run.smithi069.stdout:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:31:59.218 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-21T04:31:59.219 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-21T04:31:59.233 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 396a1cf2-e196-4a31-b42f-998c792d93aa... 2024-01-21T04:31:59.234 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-21T04:31:59.276 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-21T04:31:59.324 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-01-21T04:31:59.324 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-01-21T04:31:59.324 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-21T04:31:59.324 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-01-21T04:31:59.324 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.324 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.325 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-01-21T04:31:59.326 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790],172.21.15.144" 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: mon client directed command retry = 5 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: mon election default strategy = 1 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: ms inject socket failures = 5000 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: ms type = async 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: osd heartbeat grace = 80 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-01-21T04:31:59.327 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.328 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-01-21T04:31:59.329 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore zero block detection = True 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: debug monc = 20 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 10 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: osd class default list = * 2024-01-21T04:31:59.330 INFO:teuthology.orchestra.run.smithi069.stdout: osd class load list = * 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: osd client watch timeout = 120 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify cached snaps = True 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify missing on start = True 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: osd fast shutdown = False 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock override recovery settings = True 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: osd memory target = 1610612736 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: osd op queue = debug_random 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: osd op queue cut off = debug_random 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: osd shutdown pgref assert = True 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-01-21T04:31:59.331 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-01-21T04:31:59.332 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-01-21T04:31:59.332 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-01-21T04:31:59.332 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-01-21T04:31:59.332 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-01-21T04:31:59.332 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-01-21T04:31:59.332 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-01-21T04:31:59.332 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.332 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-01-21T04:31:59.332 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-01-21T04:31:59.332 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-01-21T04:31:59.332 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.333 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: mon min osdmap epochs = 50 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd crush smoke test = False 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune interval = 2 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune min = 15 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: mon osdmap full prune txsize = 2 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: mon scrub interval = 300 2024-01-21T04:31:59.334 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on pool no app = False 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout: paxos service trim min = 10 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout: debug monc = 20 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout: debug objecter = 20 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout: debug rados = 20 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-01-21T04:31:59.335 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-01-21T04:31:59.336 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-01-21T04:31:59.336 INFO:teuthology.orchestra.run.smithi144.stdout:[global] 2024-01-21T04:31:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: chdir = "" 2024-01-21T04:31:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-21T04:31:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: auth supported = cephx 2024-01-21T04:31:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: filestore xattr use omap = true 2024-01-21T04:31:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.336 INFO:teuthology.orchestra.run.smithi144.stdout: mon clock drift allowed = 1.000 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: osd crush chooseleaf type = 0 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: auth debug = true 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on old message = true 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: ms die on bug = true 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: mon pg warn max object skew = 0 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: # disable pg_autoscaler by default for new pools 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-21T04:31:59.337 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default size = 2 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow primary affinity = true 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd allow pg remap = true 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on legacy crush tunables = false 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on crush straw calc version zero = false 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on no sortbitwise = false 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on osd down out interval zero = false 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on too few osds = false 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: mon_allow_pool_size_one = true 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.338 INFO:teuthology.orchestra.run.smithi144.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: osd default data pool replay window = 5 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: mon allow pool delete = true 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: mon cluster log file level = debug 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: debug asserts on shutdown = true 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: mon health detail to clog = false 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: mon host = "172.21.15.69,[v2:172.21.15.69:3301,v1:172.21.15.69:6790],172.21.15.144" 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: mon client directed command retry = 5 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: mon election default strategy = 1 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: ms inject socket failures = 5000 2024-01-21T04:31:59.339 INFO:teuthology.orchestra.run.smithi144.stdout: ms type = async 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: osd heartbeat grace = 80 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: fsid = 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout:[osd] 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: osd journal size = 100 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub load threshold = 5.0 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: osd scrub max interval = 600 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: osd mclock profile = high_recovery_ops 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: osd recover clone overlap = true 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: osd recovery max chunk = 1048576 2024-01-21T04:31:59.340 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug shutdown = true 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug op order = true 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify stray on activate = true 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug trim objects = true 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: osd open classes on start = true 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug pg log writeout = true 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: osd deep scrub update digest min age = 30 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: osd map max advance = 10 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: journal zero on create = true 2024-01-21T04:31:59.341 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: filestore ondisk finisher threads = 3 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: filestore apply finisher threads = 3 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: bdev debug aio = true 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug misdirected ops = true 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: bdev async discard = True 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: bdev enable discard = True 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore block size = 96636764160 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore fsck on mount = True 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: bluestore zero block detection = True 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: debug bluefs = 1 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: debug bluestore = 1 2024-01-21T04:31:59.342 INFO:teuthology.orchestra.run.smithi144.stdout: debug monc = 20 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: debug osd = 20 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: debug rocksdb = 10 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd backfillfull_ratio = 0.85 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd full ratio = 0.9 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd nearfull ratio = 0.8 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: osd class default list = * 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: osd class load list = * 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: osd client watch timeout = 120 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify cached snaps = True 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: osd debug verify missing on start = True 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: osd failsafe full ratio = 0.95 2024-01-21T04:31:59.343 INFO:teuthology.orchestra.run.smithi144.stdout: osd fast shutdown = False 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: osd mclock override recovery settings = True 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: osd memory target = 1610612736 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: osd objectstore = bluestore 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: osd op queue = debug_random 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: osd op queue cut off = debug_random 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: osd shutdown pgref assert = True 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr] 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: debug mgr = 20 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2024-01-21T04:31:59.344 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: mgr/telemetry/nag = false 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout:[mon] 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: debug mon = 20 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: debug paxos = 20 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: debug auth = 20 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: mon data avail warn = 5 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: mon mgr mkfs grace = 240 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min pgs per osd = 4 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd reporter subtree level = osd 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd prime pg temp = true 2024-01-21T04:31:59.345 INFO:teuthology.orchestra.run.smithi144.stdout: mon reweight min bytes per osd = 10 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: auth mon ticket ttl = 660 # 11m 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: auth service ticket ttl = 240 # 4m 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: # don't complain about insecure global_id in the test suite 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: # 1m isn't quite enough 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: mon_down_mkfs_grace = 2m 2024-01-21T04:31:59.346 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: mon_warn_on_filestore_osds = false 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: mon min osdmap epochs = 50 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: mon osd crush smoke test = False 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: mon osdmap full prune interval = 2 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: mon osdmap full prune min = 15 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: mon osdmap full prune txsize = 2 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: mon scrub interval = 300 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: mon warn on pool no app = False 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: paxos service trim min = 10 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout:[client] 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: rgw cache enabled = true 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable ops log = true 2024-01-21T04:31:59.347 INFO:teuthology.orchestra.run.smithi144.stdout: rgw enable usage log = true 2024-01-21T04:31:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-21T04:31:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-21T04:31:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: debug monc = 20 2024-01-21T04:31:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: debug ms = 1 2024-01-21T04:31:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: debug objecter = 20 2024-01-21T04:31:59.348 INFO:teuthology.orchestra.run.smithi144.stdout: debug rados = 20 2024-01-21T04:31:59.348 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.a] 2024-01-21T04:31:59.348 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.c] 2024-01-21T04:31:59.348 INFO:teuthology.orchestra.run.smithi144.stdout:[mon.b] 2024-01-21T04:31:59.349 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-21T04:31:59.350 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-21T04:31:59.445 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-21T04:31:59.445 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:31:59.445 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-21T04:31:59.459 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:31:59.459 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-21T04:31:59.514 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2024-01-21T04:31:59.514 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:31:59.514 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-21T04:31:59.514 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-21T04:31:59.594 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:31:59.594 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:31:59.648 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi144.front.sepia.ceph.com 2024-01-21T04:31:59.648 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:31:59.649 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-21T04:31:59.649 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-21T04:31:59.688 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:31:59.688 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:31:59.743 INFO:tasks.ceph:Setting up mon nodes... 2024-01-21T04:31:59.743 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-21T04:31:59.744 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-21T04:31:59.798 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-21T04:31:59.800 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-21T04:31:59.864 INFO:teuthology.orchestra.run.smithi144.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-21T04:31:59.866 INFO:tasks.ceph:Setting up mds nodes... 2024-01-21T04:31:59.866 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-21T04:31:59.867 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-21T04:31:59.908 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-21T04:31:59.924 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-21T04:31:59.966 INFO:teuthology.orchestra.run.smithi144.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-21T04:31:59.981 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-21T04:31:59.981 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi144.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-21T04:31:59.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-21T04:32:00.007 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-21T04:32:00.007 INFO:tasks.ceph:role: osd.0 2024-01-21T04:32:00.008 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-21T04:32:00.008 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-21T04:32:00.082 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T04:32:00.082 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T04:32:00.082 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T04:32:00.082 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T04:32:00.082 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T04:32:00.082 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-01-21T04:32:00.082 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T04:32:00.082 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T04:32:00.082 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T04:32:00.082 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T04:32:00.563 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-01-21T04:32:00.565 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-01-21T04:32:00.565 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-21T04:32:00.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-21T04:32:00.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-21T04:32:00.773 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-21T04:32:00.773 INFO:tasks.ceph:role: osd.1 2024-01-21T04:32:00.773 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-21T04:32:00.773 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-21T04:32:00.847 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T04:32:00.848 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T04:32:00.848 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T04:32:00.848 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T04:32:00.848 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T04:32:00.848 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-01-21T04:32:00.848 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T04:32:00.848 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T04:32:00.848 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T04:32:00.848 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T04:32:01.300 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-01-21T04:32:01.301 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-01-21T04:32:01.302 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-21T04:32:01.374 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-21T04:32:01.445 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-21T04:32:01.511 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-21T04:32:01.511 INFO:tasks.ceph:role: osd.2 2024-01-21T04:32:01.511 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-21T04:32:01.512 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-21T04:32:01.585 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T04:32:01.585 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T04:32:01.585 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T04:32:01.585 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T04:32:01.586 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T04:32:01.586 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-01-21T04:32:01.586 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T04:32:01.586 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T04:32:01.586 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T04:32:01.586 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T04:32:02.080 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-01-21T04:32:02.082 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-01-21T04:32:02.082 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-21T04:32:02.177 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-21T04:32:02.210 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-21T04:32:02.278 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-21T04:32:02.278 INFO:tasks.ceph:role: osd.3 2024-01-21T04:32:02.278 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-21T04:32:02.278 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-21T04:32:02.354 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T04:32:02.354 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T04:32:02.354 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T04:32:02.354 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T04:32:02.354 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T04:32:02.354 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-01-21T04:32:02.354 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T04:32:02.355 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T04:32:02.355 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T04:32:02.355 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T04:32:02.868 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-01-21T04:32:02.870 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-01-21T04:32:02.870 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-21T04:32:02.999 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-21T04:32:03.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:32:03.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:03.127+0000 7f0c223e24c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-21T04:32:03.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:03.130+0000 7f0c223e24c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-21T04:32:03.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:03.130+0000 7f0c223e24c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-21T04:32:03.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:03.130+0000 7f0c223e24c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-21T04:32:03.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:03.130+0000 7f0c223e24c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-21T04:32:03.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:03.137+0000 7f0c223e24c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-21T04:32:05.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-21T04:32:06.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:32:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:06.109+0000 7ff2201be4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-21T04:32:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:06.110+0000 7ff2201be4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-21T04:32:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:06.110+0000 7ff2201be4c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-21T04:32:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:06.110+0000 7ff2201be4c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-21T04:32:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:06.110+0000 7ff2201be4c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-21T04:32:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:06.110+0000 7ff2201be4c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-21T04:32:08.955 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-21T04:32:08.986 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:32:09.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:09.079+0000 7efdc46144c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-21T04:32:09.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:09.079+0000 7efdc46144c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-21T04:32:09.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:09.079+0000 7efdc46144c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-21T04:32:09.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:09.079+0000 7efdc46144c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-21T04:32:09.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:09.079+0000 7efdc46144c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-21T04:32:09.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:09.080+0000 7efdc46144c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-21T04:32:11.931 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-21T04:32:11.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:32:12.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:12.052+0000 7f4da5a0c4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-21T04:32:12.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:12.052+0000 7f4da5a0c4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-21T04:32:12.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:12.052+0000 7f4da5a0c4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-21T04:32:12.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:12.052+0000 7f4da5a0c4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-21T04:32:12.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:12.052+0000 7f4da5a0c4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-21T04:32:12.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:12.053+0000 7f4da5a0c4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-21T04:32:14.893 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-21T04:32:14.924 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-21T04:32:14.956 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-21T04:32:14.956 INFO:tasks.ceph:role: osd.4 2024-01-21T04:32:14.956 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com 2024-01-21T04:32:14.957 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-21T04:32:15.034 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T04:32:15.035 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T04:32:15.035 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T04:32:15.035 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T04:32:15.035 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T04:32:15.035 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-01-21T04:32:15.035 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T04:32:15.035 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T04:32:15.035 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T04:32:15.035 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T04:32:15.558 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-01-21T04:32:15.561 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-01-21T04:32:15.561 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-21T04:32:15.716 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-21T04:32:15.753 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-21T04:32:15.823 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-21T04:32:15.823 INFO:tasks.ceph:role: osd.5 2024-01-21T04:32:15.823 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com 2024-01-21T04:32:15.823 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-21T04:32:15.900 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T04:32:15.900 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T04:32:15.900 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T04:32:15.900 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T04:32:15.900 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T04:32:15.900 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-01-21T04:32:15.900 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T04:32:15.900 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T04:32:15.900 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T04:32:15.900 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T04:32:16.357 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-01-21T04:32:16.360 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-01-21T04:32:16.360 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-21T04:32:16.455 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-21T04:32:16.493 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-21T04:32:16.562 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-21T04:32:16.562 INFO:tasks.ceph:role: osd.6 2024-01-21T04:32:16.562 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com 2024-01-21T04:32:16.563 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-21T04:32:16.638 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T04:32:16.639 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T04:32:16.639 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T04:32:16.639 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T04:32:16.639 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T04:32:16.639 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-01-21T04:32:16.639 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T04:32:16.640 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T04:32:16.640 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T04:32:16.640 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T04:32:17.129 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-01-21T04:32:17.131 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-01-21T04:32:17.131 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-21T04:32:17.257 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-21T04:32:17.294 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-21T04:32:17.364 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-21T04:32:17.364 INFO:tasks.ceph:role: osd.7 2024-01-21T04:32:17.365 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi144.front.sepia.ceph.com 2024-01-21T04:32:17.365 DEBUG:teuthology.orchestra.run.smithi144:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-21T04:32:17.442 INFO:teuthology.orchestra.run.smithi144.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-21T04:32:17.442 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-21T04:32:17.442 INFO:teuthology.orchestra.run.smithi144.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-21T04:32:17.442 INFO:teuthology.orchestra.run.smithi144.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-21T04:32:17.442 INFO:teuthology.orchestra.run.smithi144.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-21T04:32:17.442 INFO:teuthology.orchestra.run.smithi144.stdout: = sunit=0 swidth=0 blks 2024-01-21T04:32:17.442 INFO:teuthology.orchestra.run.smithi144.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-21T04:32:17.442 INFO:teuthology.orchestra.run.smithi144.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-21T04:32:17.442 INFO:teuthology.orchestra.run.smithi144.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-21T04:32:17.442 INFO:teuthology.orchestra.run.smithi144.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-21T04:32:17.936 INFO:teuthology.orchestra.run.smithi144.stdout:Discarding blocks...Done. 2024-01-21T04:32:17.938 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi144.front.sepia.ceph.com -o noatime 2024-01-21T04:32:17.938 DEBUG:teuthology.orchestra.run.smithi144:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-21T04:32:18.073 DEBUG:teuthology.orchestra.run.smithi144:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-21T04:32:18.110 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:32:18.218 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:18.217+0000 7f63cf6144c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-21T04:32:18.219 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:18.217+0000 7f63cf6144c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-21T04:32:18.219 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:18.217+0000 7f63cf6144c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-21T04:32:18.219 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:18.217+0000 7f63cf6144c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-21T04:32:18.219 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:18.217+0000 7f63cf6144c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-21T04:32:18.219 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:18.218+0000 7f63cf6144c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-21T04:32:21.059 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-21T04:32:21.092 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:32:21.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:21.183+0000 7fe5f57374c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-21T04:32:21.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:21.183+0000 7fe5f57374c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-21T04:32:21.184 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:21.183+0000 7fe5f57374c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-21T04:32:21.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:21.183+0000 7fe5f57374c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-21T04:32:21.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:21.183+0000 7fe5f57374c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-21T04:32:21.185 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:21.184+0000 7fe5f57374c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-21T04:32:24.027 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-21T04:32:24.058 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:32:24.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:24.149+0000 7fe28dc8a4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-21T04:32:24.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:24.149+0000 7fe28dc8a4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-21T04:32:24.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:24.149+0000 7fe28dc8a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-21T04:32:24.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:24.149+0000 7fe28dc8a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-21T04:32:24.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:24.149+0000 7fe28dc8a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-21T04:32:24.150 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:24.150+0000 7fe28dc8a4c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-21T04:32:26.991 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-21T04:32:27.022 DEBUG:teuthology.orchestra.run.smithi144:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:32:27.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:27.112+0000 7f10c8c254c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-21T04:32:27.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:27.112+0000 7f10c8c254c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-21T04:32:27.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:27.112+0000 7f10c8c254c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-21T04:32:27.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:27.112+0000 7f10c8c254c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-21T04:32:27.113 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:27.112+0000 7f10c8c254c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-21T04:32:27.114 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:27.113+0000 7f10c8c254c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-21T04:32:29.956 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-21T04:32:29.987 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-21T04:32:29.987 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:32:29.988 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-21T04:32:30.019 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:32:30.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-21T04:32:30.090 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:32:30.090 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-21T04:32:30.159 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:32:30.160 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-21T04:32:30.230 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:32:30.230 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-21T04:32:30.299 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:32:30.300 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-21T04:32:30.331 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:32:30.331 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-21T04:32:30.402 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:32:30.402 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-21T04:32:30.475 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:32:30.476 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-21T04:32:30.545 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:32:30.545 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-21T04:32:30.613 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:32:30.613 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-21T04:32:30.632 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:32:30.632 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-21T04:32:30.668 INFO:tasks.ceph:Adding keys to all mons... 2024-01-21T04:32:30.669 DEBUG:teuthology.orchestra.run.smithi069:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-21T04:32:30.675 DEBUG:teuthology.orchestra.run.smithi144:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-21T04:32:30.712 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.y] 2024-01-21T04:32:30.712 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQA/nqxlBquTLxAAiD4/4xUeqAskBa0NDCvLHg== 2024-01-21T04:32:30.712 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2024-01-21T04:32:30.712 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBDnqxl2oavBxAAWrCgg+Ay0YepP91Huni6bA== 2024-01-21T04:32:30.712 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2024-01-21T04:32:30.712 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBGnqxlI4uaBhAAgEHlJjNmpkSy/cDk4yq9Ow== 2024-01-21T04:32:30.712 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2024-01-21T04:32:30.712 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBJnqxl+AjOBBAASFOYhjMyP7Z6EqHmYD49GA== 2024-01-21T04:32:30.712 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.3] 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBMnqxl8uwvAxAAnvphLE1xm/Hw+k5C5hb4wQ== 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.x] 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQA/nqxlX9+EMxAAbo7EL0Yz+IAGTfXKAWmcbg== 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.4] 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBSnqxl1GcEDRAABeUQmG0ZV6bHn0THtyBIEA== 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.5] 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBVnqxl8jr6ChAAoGOOjp69Ctxp+c9/F0xtLg== 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.6] 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBYnqxlXVHyCBAAEetm3AK7Yqi1W6hniHrxBQ== 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.7] 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBbnqxlQk3EBhAASnfSw0KmaxxIhW/S1VecfQ== 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-01-21T04:32:30.713 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQA/nqxleSsvNhAANab3gaToi0MXYAeJtxSqWw== 2024-01-21T04:32:30.714 INFO:teuthology.orchestra.run.smithi069.stdout:[client.1] 2024-01-21T04:32:30.714 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQA/nqxlZ9ibORAAw+NgGVmFUN8YxEv3vcwrvQ== 2024-01-21T04:32:30.735 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.y] 2024-01-21T04:32:30.735 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQA/nqxlBquTLxAAiD4/4xUeqAskBa0NDCvLHg== 2024-01-21T04:32:30.735 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.0] 2024-01-21T04:32:30.735 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBDnqxl2oavBxAAWrCgg+Ay0YepP91Huni6bA== 2024-01-21T04:32:30.735 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.1] 2024-01-21T04:32:30.735 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBGnqxlI4uaBhAAgEHlJjNmpkSy/cDk4yq9Ow== 2024-01-21T04:32:30.735 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.2] 2024-01-21T04:32:30.735 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBJnqxl+AjOBBAASFOYhjMyP7Z6EqHmYD49GA== 2024-01-21T04:32:30.735 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.3] 2024-01-21T04:32:30.735 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBMnqxl8uwvAxAAnvphLE1xm/Hw+k5C5hb4wQ== 2024-01-21T04:32:30.735 INFO:teuthology.orchestra.run.smithi144.stdout:[mgr.x] 2024-01-21T04:32:30.736 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQA/nqxlX9+EMxAAbo7EL0Yz+IAGTfXKAWmcbg== 2024-01-21T04:32:30.736 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.4] 2024-01-21T04:32:30.736 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBSnqxl1GcEDRAABeUQmG0ZV6bHn0THtyBIEA== 2024-01-21T04:32:30.736 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.5] 2024-01-21T04:32:30.736 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBVnqxl8jr6ChAAoGOOjp69Ctxp+c9/F0xtLg== 2024-01-21T04:32:30.736 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.6] 2024-01-21T04:32:30.736 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBYnqxlXVHyCBAAEetm3AK7Yqi1W6hniHrxBQ== 2024-01-21T04:32:30.736 INFO:teuthology.orchestra.run.smithi144.stdout:[osd.7] 2024-01-21T04:32:30.736 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQBbnqxlQk3EBhAASnfSw0KmaxxIhW/S1VecfQ== 2024-01-21T04:32:30.736 INFO:teuthology.orchestra.run.smithi144.stdout:[client.0] 2024-01-21T04:32:30.737 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQA/nqxleSsvNhAANab3gaToi0MXYAeJtxSqWw== 2024-01-21T04:32:30.737 INFO:teuthology.orchestra.run.smithi144.stdout:[client.1] 2024-01-21T04:32:30.737 INFO:teuthology.orchestra.run.smithi144.stdout: key = AQA/nqxlZ9ibORAAw+NgGVmFUN8YxEv3vcwrvQ== 2024-01-21T04:32:30.738 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-21T04:32:30.757 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-21T04:32:30.830 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:30.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:30.884 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:30.887 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:30.937 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:30.940 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:30.991 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:30.993 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:31.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-21T04:32:31.048 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-21T04:32:31.099 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:31.102 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:31.154 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:31.156 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:31.207 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:31.209 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:31.260 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:31.262 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-21T04:32:31.313 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-21T04:32:31.316 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-21T04:32:31.370 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-21T04:32:31.372 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-21T04:32:31.426 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-21T04:32:31.427 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-21T04:32:31.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-21T04:32:31.549 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-21T04:32:31.576 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-21T04:32:31.642 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-21T04:32:31.739 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-21T04:32:31.766 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-21T04:32:31.793 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-21T04:32:31.890 DEBUG:teuthology.orchestra.run.smithi144:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-21T04:32:31.916 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:32:31.919 DEBUG:teuthology.orchestra.run.smithi144:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-21T04:32:31.971 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-21T04:32:31.971 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-21T04:32:31.972 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-21T04:32:31.972 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-01-21T04:32:31.974 INFO:tasks.ceph.mon.a:Started 2024-01-21T04:32:31.974 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-21T04:32:31.974 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-21T04:32:31.974 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-01-21T04:32:31.976 INFO:tasks.ceph.mon.c:Started 2024-01-21T04:32:31.976 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-01-21T04:32:31.976 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-21T04:32:31.976 DEBUG:teuthology.orchestra.run.smithi144:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-01-21T04:32:32.014 INFO:tasks.ceph.mon.b:Started 2024-01-21T04:32:32.014 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-21T04:32:32.014 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-21T04:32:32.014 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-01-21T04:32:32.017 INFO:tasks.ceph.mgr.y:Started 2024-01-21T04:32:32.017 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-21T04:32:32.017 DEBUG:teuthology.orchestra.run.smithi144:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-01-21T04:32:32.019 INFO:tasks.ceph.mgr.x:Started 2024-01-21T04:32:32.019 INFO:tasks.ceph:Setting crush tunables to default 2024-01-21T04:32:32.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd crush tunables default 2024-01-21T04:32:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.118+0000 7fc4f6d61640 10 monclient: get_monmap_and_config 2024-01-21T04:32:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.118+0000 7fc4f6d61640 10 monclient: build_initial_monmap 2024-01-21T04:32:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.118+0000 7fc4f6d61640 10 monclient: monmap: 2024-01-21T04:32:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:32:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:32:32.119041+0000 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:32:32.119041+0000 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.118+0000 7fc4f6d61640 1 Processor -- start 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.118+0000 7fc4f6d61640 1 -- start start 2024-01-21T04:32:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.118+0000 7fc4f6d61640 10 monclient: init 2024-01-21T04:32:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-01-21T04:32:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4f00a8b20 addr v1:172.21.15.144:6789/0 2024-01-21T04:32:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4f00a9d10 0x7fc4f013f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 10 monclient(hunting): picked mon.noname-c con 0x7fc4f00a9d10 addr v2:172.21.15.144:3300/0 2024-01-21T04:32:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4f00b2310 addr v1:172.21.15.69:6789/0 2024-01-21T04:32:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f01456f0 con 0x7fc4f00b2310 2024-01-21T04:32:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f00a98c0 con 0x7fc4f00a8b20 2024-01-21T04:32:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:32.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f0146290 con 0x7fc4f00a9d10 2024-01-21T04:32:32.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:32.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f6d61640 10 monclient(hunting): authenticate will time out at 1310.460613s 2024-01-21T04:32:32.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f52fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7fc4f00b2310 legacy=0x7fc4f013f5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-01-21T04:32:32.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4f4afd640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fc4f00a8b20 legacy=0x7fc4f0145b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-01-21T04:32:32.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4effff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fc4f00a9d10 msgr2=0x7fc4f013f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-01-21T04:32:32.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.119+0000 7fc4effff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4f00a9d10 0x7fc4f013f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T04:32:32.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.320+0000 7fc4f52fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7fc4f00b2310 legacy=0x7fc4f013f5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-01-21T04:32:32.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.320+0000 7fc4effff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fc4f00a9d10 msgr2=0x7fc4f013f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-01-21T04:32:32.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.320+0000 7fc4f4afd640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fc4f00a8b20 legacy=0x7fc4f0145b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-01-21T04:32:32.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.320+0000 7fc4effff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4f00a9d10 0x7fc4f013f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T04:32:32.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.721+0000 7fc4f4afd640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fc4f00a8b20 legacy=0x7fc4f0145b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-01-21T04:32:32.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.721+0000 7fc4f52fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7fc4f00b2310 legacy=0x7fc4f013f5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-01-21T04:32:32.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.721+0000 7fc4effff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fc4f00a9d10 msgr2=0x7fc4f013f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-01-21T04:32:32.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:32.721+0000 7fc4effff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4f00a9d10 0x7fc4f013f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T04:32:33.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:33.522+0000 7fc4f52fe640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7fc4f00b2310 legacy=0x7fc4f013f5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.69:6789/0 2024-01-21T04:32:33.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:33.522+0000 7fc4effff640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fc4f00a9d10 msgr2=0x7fc4f013f080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.144:3300/0 2024-01-21T04:32:33.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:33.522+0000 7fc4f4afd640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fc4f00a8b20 legacy=0x7fc4f0145b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-01-21T04:32:33.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:33.522+0000 7fc4effff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4f00a9d10 0x7fc4f013f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T04:32:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 10 monclient(hunting): tick 2024-01-21T04:32:35.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 10 monclient(hunting): _check_auth_tickets 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 1 monclient(hunting): continuing hunt 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 10 monclient(hunting): _reopen_session rank -1 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 1 -- >> v2:172.21.15.144:3300/0 conn(0x7fc4f00a9d10 msgr2=0x7fc4f013f080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4f00a9d10 0x7fc4f013f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fc4f00a8b20 legacy=0x7fc4f0145b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 1 -- >> v1:172.21.15.69:6789/0 conn(0x7fc4f00b2310 legacy=0x7fc4f013f5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 10 monclient: _add_conns ranks=[3,2] 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc0036a0 0x7fc4dc005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc4dc0036a0 addr v2:172.21.15.69:3300/0 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006110 0x7fc4dc0085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc4dc006110 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f0146290 con 0x7fc4dc0036a0 2024-01-21T04:32:35.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:35.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc008ca0 con 0x7fc4dc006110 2024-01-21T04:32:35.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4ee7fc640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:35.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4f4afd640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc0036a0 msgr2=0x7fc4dc005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2024-01-21T04:32:35.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4effff640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006110 msgr2=0x7fc4dc0085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2024-01-21T04:32:35.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4f4afd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc0036a0 0x7fc4dc005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T04:32:35.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.119+0000 7fc4effff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006110 0x7fc4dc0085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T04:32:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.320+0000 7fc4f4afd640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc0036a0 msgr2=0x7fc4dc005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2024-01-21T04:32:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.320+0000 7fc4f4afd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc0036a0 0x7fc4dc005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T04:32:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.320+0000 7fc4effff640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006110 msgr2=0x7fc4dc0085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2024-01-21T04:32:35.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.320+0000 7fc4effff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006110 0x7fc4dc0085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T04:32:35.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.721+0000 7fc4f4afd640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc0036a0 msgr2=0x7fc4dc005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2024-01-21T04:32:35.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.721+0000 7fc4effff640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006110 msgr2=0x7fc4dc0085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2024-01-21T04:32:35.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.721+0000 7fc4f4afd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc0036a0 0x7fc4dc005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T04:32:35.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:35.721+0000 7fc4effff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006110 0x7fc4dc0085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T04:32:36.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:36.522+0000 7fc4f4afd640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc0036a0 msgr2=0x7fc4dc005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2024-01-21T04:32:36.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:36.522+0000 7fc4effff640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006110 msgr2=0x7fc4dc0085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2024-01-21T04:32:36.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:36.522+0000 7fc4f4afd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc0036a0 0x7fc4dc005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T04:32:36.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:36.522+0000 7fc4effff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006110 0x7fc4dc0085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T04:32:38.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 10 monclient(hunting): tick 2024-01-21T04:32:38.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 10 monclient(hunting): _check_auth_tickets 2024-01-21T04:32:38.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 1 monclient(hunting): continuing hunt 2024-01-21T04:32:38.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 10 monclient(hunting): _reopen_session rank -1 2024-01-21T04:32:38.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006110 msgr2=0x7fc4dc0085d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T04:32:38.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006110 0x7fc4dc0085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:38.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc0036a0 msgr2=0x7fc4dc005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T04:32:38.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc0036a0 0x7fc4dc005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4f52fe640 1 -- reap_dead start 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc003b10 0x7fc4dc0065c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc4dc003b10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc006b00 0x7fc4dc006f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fc4dc006b00 addr v2:172.21.15.69:3300/0 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4dc007480 addr v1:172.21.15.144:6789/0 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f01463d0 con 0x7fc4dc007480 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f0146290 con 0x7fc4dc006b00 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc008ca0 con 0x7fc4dc003b10 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4ee7fc640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:38.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4effff640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc006b00 msgr2=0x7fc4dc006f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2024-01-21T04:32:38.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4f4afd640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc003b10 msgr2=0x7fc4dc0065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2024-01-21T04:32:38.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4f52fe640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fc4dc007480 legacy=0x7fc4dc007900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-01-21T04:32:38.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4f4afd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc003b10 0x7fc4dc0065c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T04:32:38.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.119+0000 7fc4effff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc006b00 0x7fc4dc006f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T04:32:38.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.320+0000 7fc4f4afd640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc003b10 msgr2=0x7fc4dc0065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2024-01-21T04:32:38.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.320+0000 7fc4effff640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc006b00 msgr2=0x7fc4dc006f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2024-01-21T04:32:38.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.320+0000 7fc4f4afd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc003b10 0x7fc4dc0065c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T04:32:38.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.320+0000 7fc4effff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc006b00 0x7fc4dc006f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T04:32:38.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.320+0000 7fc4f52fe640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fc4dc007480 legacy=0x7fc4dc007900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-01-21T04:32:38.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.721+0000 7fc4f4afd640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc003b10 msgr2=0x7fc4dc0065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2024-01-21T04:32:38.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.721+0000 7fc4effff640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc006b00 msgr2=0x7fc4dc006f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2024-01-21T04:32:38.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.721+0000 7fc4f4afd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc003b10 0x7fc4dc0065c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T04:32:38.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.721+0000 7fc4effff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc006b00 0x7fc4dc006f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T04:32:38.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:38.721+0000 7fc4f52fe640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fc4dc007480 legacy=0x7fc4dc007900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-01-21T04:32:38.925 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:00:06.801 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:32:39.212 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:00:07.118 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:32:39.213 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:00:07.120 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:32:39.233 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:00:07.108 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:32:39.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:39.522+0000 7fc4effff640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc006b00 msgr2=0x7fc4dc006f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3300/0 2024-01-21T04:32:39.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:39.522+0000 7fc4f4afd640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc003b10 msgr2=0x7fc4dc0065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.69:3301/0 2024-01-21T04:32:39.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:39.522+0000 7fc4effff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc006b00 0x7fc4dc006f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T04:32:39.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:39.522+0000 7fc4f4afd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc003b10 0x7fc4dc0065c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T04:32:39.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:39.522+0000 7fc4f52fe640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fc4dc007480 legacy=0x7fc4dc007900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.144:6789/0 2024-01-21T04:32:39.531 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:00:07.438 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:32:39.533 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:00:07.440 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 10 monclient(hunting): tick 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 10 monclient(hunting): _check_auth_tickets 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 1 monclient(hunting): continuing hunt 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 10 monclient(hunting): _reopen_session rank -1 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 1 -- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc003b10 msgr2=0x7fc4dc0065c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc003b10 0x7fc4dc0065c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 1 -- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc006b00 msgr2=0x7fc4dc006f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4dc006b00 0x7fc4dc006f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 1 -- >> v1:172.21.15.144:6789/0 conn(0x7fc4dc007480 legacy=0x7fc4dc007900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 10 monclient: _add_conns ranks=[0,4] 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4dc0084b0 addr v1:172.21.15.69:6789/0 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fc4dc0116c0 addr v2:172.21.15.144:3300/0 2024-01-21T04:32:41.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f01456f0 con 0x7fc4dc0084b0 2024-01-21T04:32:41.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:41.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f0146290 con 0x7fc4dc0116c0 2024-01-21T04:32:41.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.119+0000 7fc4ee7fc640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:41.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.173+0000 7fc4effff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:41.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.189+0000 7fc4effff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:54270/0 (socket says 172.21.15.69:54270) 2024-01-21T04:32:41.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.189+0000 7fc4effff640 1 -- 172.21.15.69:0/3935418302 learned_addr learned my addr 172.21.15.69:0/3935418302 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:32:41.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.189+0000 7fc4effff640 10 monclient(hunting): get_auth_request con 0x7fc4dc0116c0 auth_method 0 2024-01-21T04:32:41.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.189+0000 7fc4effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:41.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.189+0000 7fc4effff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:41.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.189+0000 7fc4effff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:41.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.208+0000 7fc4effff640 1 -- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 msgr2=0x7fc4dc013ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-21T04:32:41.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.208+0000 7fc4effff640 1 -- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 msgr2=0x7fc4dc013ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T04:32:41.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.208+0000 7fc4effff640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T04:32:41.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.208+0000 7fc4effff640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T04:32:41.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.588+0000 7fc4effff640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:41.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.588+0000 7fc4effff640 10 monclient(hunting): get_auth_request con 0x7fc4dc0116c0 auth_method 0 2024-01-21T04:32:41.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.588+0000 7fc4effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:41.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.588+0000 7fc4effff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:41.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.588+0000 7fc4effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T04:32:41.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.589+0000 7fc4effff640 1 -- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 msgr2=0x7fc4dc013ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-21T04:32:41.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.589+0000 7fc4effff640 1 -- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 msgr2=0x7fc4dc013ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T04:32:41.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.589+0000 7fc4effff640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T04:32:41.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:41.589+0000 7fc4effff640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-21T04:32:42.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.028+0000 7fc4effff640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:42.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.039+0000 7fc4effff640 10 monclient(hunting): get_auth_request con 0x7fc4dc0116c0 auth_method 0 2024-01-21T04:32:42.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.039+0000 7fc4effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:42.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.039+0000 7fc4effff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:42.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.039+0000 7fc4effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T04:32:42.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.046+0000 7fc4effff640 1 -- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 msgr2=0x7fc4dc013ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-21T04:32:42.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.046+0000 7fc4effff640 1 -- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 msgr2=0x7fc4dc013ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T04:32:42.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.046+0000 7fc4effff640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T04:32:42.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.046+0000 7fc4effff640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-21T04:32:42.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.848+0000 7fc4effff640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:42.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.848+0000 7fc4effff640 10 monclient(hunting): get_auth_request con 0x7fc4dc0116c0 auth_method 0 2024-01-21T04:32:42.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.848+0000 7fc4effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:42.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.848+0000 7fc4effff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:42.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.848+0000 7fc4effff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T04:32:42.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.849+0000 7fc4effff640 1 -- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 msgr2=0x7fc4dc013ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-21T04:32:42.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.849+0000 7fc4effff640 1 -- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 msgr2=0x7fc4dc013ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T04:32:42.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.849+0000 7fc4effff640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T04:32:42.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:42.849+0000 7fc4effff640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-21T04:32:44.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 10 monclient(hunting): tick 2024-01-21T04:32:44.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 10 monclient(hunting): _check_auth_tickets 2024-01-21T04:32:44.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 1 monclient(hunting): continuing hunt 2024-01-21T04:32:44.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 10 monclient(hunting): _reopen_session rank -1 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 1 -- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 msgr2=0x7fc4dc013ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 1 --2- 172.21.15.69:0/3935418302 >> v2:172.21.15.144:3300/0 conn(0x7fc4dc0116c0 0x7fc4dc013ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 1 -- 172.21.15.69:0/3935418302 >> v1:172.21.15.69:6789/0 conn(0x7fc4dc0084b0 legacy=0x7fc4dc00ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4f52fe640 1 -- 172.21.15.69:0/3935418302 reap_dead start 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 0x7fc4dc007900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc4dc006f40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4dc005af0 addr v1:172.21.15.69:6789/0 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4dc013ab0 addr v1:172.21.15.144:6789/0 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 1 -- 172.21.15.69:0/3935418302 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f01463d0 con 0x7fc4dc005af0 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 1 -- 172.21.15.69:0/3935418302 --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f00a98c0 con 0x7fc4dc013ab0 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 1 -- 172.21.15.69:0/3935418302 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f0146290 con 0x7fc4dc006f40 2024-01-21T04:32:44.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.120+0000 7fc4ee7fc640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:44.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.158+0000 7fc4f4afd640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 0x7fc4dc007900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:44.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.159+0000 7fc4f4afd640 10 monclient(hunting): get_auth_request con 0x7fc4dc006f40 auth_method 0 2024-01-21T04:32:44.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.159+0000 7fc4f4afd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:44.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.159+0000 7fc4f4afd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:44.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.159+0000 7fc4f4afd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:44.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.299+0000 7fc4f4afd640 1 -- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 msgr2=0x7fc4dc007900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-21T04:32:44.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.299+0000 7fc4f4afd640 1 -- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 msgr2=0x7fc4dc007900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T04:32:44.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.299+0000 7fc4f4afd640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 0x7fc4dc007900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T04:32:44.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.299+0000 7fc4f4afd640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 0x7fc4dc007900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T04:32:44.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.465+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784501968 0 0) 0x7fc4f01463d0 con 0x7fc4dc005af0 2024-01-21T04:32:44.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.465+0000 7fc4ef7fe640 10 monclient(hunting): _negotiate 2024-01-21T04:32:44.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.465+0000 7fc4ef7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:44.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.465+0000 7fc4ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:44.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.465+0000 7fc4ef7fe640 10 monclient(hunting): my global_id is 4104 2024-01-21T04:32:44.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.465+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d8003520 con 0x7fc4dc005af0 2024-01-21T04:32:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.500+0000 7fc4f4afd640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 0x7fc4dc007900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.500+0000 7fc4f4afd640 10 monclient(hunting): get_auth_request con 0x7fc4dc006f40 auth_method 0 2024-01-21T04:32:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.500+0000 7fc4f4afd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.500+0000 7fc4f4afd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.500+0000 7fc4f4afd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T04:32:44.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.505+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218653431 0 0) 0x7fc4f00a98c0 con 0x7fc4dc013ab0 2024-01-21T04:32:44.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.505+0000 7fc4ef7fe640 10 monclient(hunting): _negotiate 2024-01-21T04:32:44.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.505+0000 7fc4ef7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:44.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.505+0000 7fc4ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:44.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.505+0000 7fc4ef7fe640 10 monclient(hunting): my global_id is 4099 2024-01-21T04:32:44.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.505+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4f01463d0 con 0x7fc4dc013ab0 2024-01-21T04:32:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.507+0000 7fc4f4afd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.507+0000 7fc4f4afd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.507+0000 7fc4f4afd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:44.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1079180610 0 0) 0x7fc4d8003520 con 0x7fc4dc005af0 2024-01-21T04:32:44.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 msgr2=0x7fc4dc007900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:44.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 0x7fc4dc007900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:44.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 >> v1:172.21.15.144:6789/0 conn(0x7fc4dc013ab0 legacy=0x7fc4dc014070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 10 monclient: _finish_hunting 0 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 1 monclient: found mon.noname-a-legacy 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.69:6789/0 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc007e40 con 0x7fc4dc005af0 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 10 monclient: _finish_auth 0 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 10 monclient: _check_auth_tickets 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 125097436 0 0) 0x7fc4e4003e90 con 0x7fc4dc005af0 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4f6d61640 5 monclient: authenticate success, global_id 4104 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4f6d61640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 10 monclient: dump: 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:32:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] but i'm connected to v1:172.21.15.69:6789/0, reconnecting 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 >> v1:172.21.15.69:6789/0 conn(0x7fc4dc005af0 legacy=0x7fc4dc0065c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4d8003b30 0x7fc4d8003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.526+0000 7fc4ef7fe640 10 monclient(hunting): picked mon.a con 0x7fc4d8003b30 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:44.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 0x7fc4d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 10 monclient(hunting): picked mon.c con 0x7fc4dc006f40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d8004d00 0x7fc4d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 10 monclient(hunting): picked mon.b con 0x7fc4d8004d00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f0146290 con 0x7fc4d8003b30 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d800ba80 con 0x7fc4d8004d00 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d800bc40 con 0x7fc4dc006f40 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4ef7fe640 10 monclient(hunting): _finish_auth 0 2024-01-21T04:32:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.527+0000 7fc4f6d61640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-01-21T04:32:44.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.546+0000 7fc4f4afd640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4d8003b30 0x7fc4d8003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:44.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.547+0000 7fc4f4afd640 10 monclient(hunting): get_auth_request con 0x7fc4d8003b30 auth_method 0 2024-01-21T04:32:44.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.547+0000 7fc4f4afd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:44.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.547+0000 7fc4f4afd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:44.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.547+0000 7fc4f4afd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T04:32:44.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.552+0000 7fc4f4afd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:44.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.552+0000 7fc4f4afd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:44.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.552+0000 7fc4f4afd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T04:32:44.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 10 monclient(hunting): handle_auth_done global_id 4104 payload 293 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 1 -- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 msgr2=0x7fc4d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4dc006f40 0x7fc4d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 1 -- 172.21.15.69:0/3935418302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d8004d00 msgr2=0x7fc4d8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4d8004d00 0x7fc4d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 10 monclient: _finish_hunting 0 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 1 monclient: found mon.a 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 1 -- 172.21.15.69:0/3935418302 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d8003520 con 0x7fc4d8003b30 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 10 monclient: _finish_auth 0 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 10 monclient: _check_auth_tickets 2024-01-21T04:32:44.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.562+0000 7fc4f4afd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:44.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.574+0000 7fc4f4afd640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4d8003b30 0x7fc4d8003f30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0078b0 tx=0x7fc4d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfb0a79f8edaca31 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:44.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e802e6b0 con 0x7fc4d8003b30 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4ef7fe640 10 monclient: dump: 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:44.673 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e802e810 con 0x7fc4d8003b30 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4f6d61640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4ef7fe640 1 -- 172.21.15.69:0/3935418302 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e802eb80 con 0x7fc4d8003b30 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4f6d61640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4f6d61640 4 set_mon_vals no callback set 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.671+0000 7fc4ef7fe640 10 monclient: dump: 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:44.674 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:44.675 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:44.675 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:44.675 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:44.675 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:44.675 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:44.675 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:44.675 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:44.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.672+0000 7fc4f6d61640 10 monclient: shutdown 2024-01-21T04:32:44.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.672+0000 7fc4f6d61640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:44.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.672+0000 7fc4f6d61640 1 -- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4d8003b30 msgr2=0x7fc4d8003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:44.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.672+0000 7fc4f6d61640 1 --2- 172.21.15.69:0/3935418302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4d8003b30 0x7fc4d8003f30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0078b0 tx=0x7fc4d80057f0 comp rx=0 tx=0).stop 2024-01-21T04:32:44.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.672+0000 7fc4f52fe640 1 -- 172.21.15.69:0/3935418302 reap_dead start 2024-01-21T04:32:44.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.672+0000 7fc4f6d61640 1 -- 172.21.15.69:0/3935418302 shutdown_connections 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.672+0000 7fc4f6d61640 1 -- 172.21.15.69:0/3935418302 >> 172.21.15.69:0/3935418302 conn(0x7fc4f0066fb0 msgr2=0x7fc4f00b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.672+0000 7fc4f6d61640 1 -- 172.21.15.69:0/3935418302 shutdown_connections 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.672+0000 7fc4f6d61640 1 -- 172.21.15.69:0/3935418302 wait complete. 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.673+0000 7fc4f6d61640 10 monclient: build_initial_monmap 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.673+0000 7fc4f6d61640 10 monclient: monmap: 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:44.678 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.673+0000 7fc4f6d61640 1 librados: starting msgr at 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.673+0000 7fc4f6d61640 1 librados: starting objecter 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.673+0000 7fc4f6d61640 1 Processor -- start 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.673+0000 7fc4f6d61640 1 -- start start 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.673+0000 7fc4f6d61640 1 librados: setting wanted keys 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.673+0000 7fc4f6d61640 1 librados: calling monclient init 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.673+0000 7fc4f6d61640 10 monclient: init 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4f0143bb0 0x7fc4f00ab910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 10 monclient(hunting): picked mon.noname-b con 0x7fc4f0143bb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f01441e0 0x7fc4f01da860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:44.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 10 monclient(hunting): picked mon.noname-a con 0x7fc4f01441e0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:44.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4f01dada0 0x7fc4f01d6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:44.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 10 monclient(hunting): picked mon.noname-c con 0x7fc4f01dada0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:44.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:44.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f01db200 con 0x7fc4f01441e0 2024-01-21T04:32:44.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:44.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f0143f90 con 0x7fc4f0143bb0 2024-01-21T04:32:44.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:44.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f01da3d0 con 0x7fc4f01dada0 2024-01-21T04:32:44.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.674+0000 7fc4f6d61640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:44.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.675+0000 7fc4f6d61640 10 monclient(hunting): authenticate will time out at 1323.016207s 2024-01-21T04:32:44.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.680+0000 7fc4effff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f01441e0 0x7fc4f01da860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:44.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.682+0000 7fc4effff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f01441e0 0x7fc4f01da860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55330/0 (socket says 172.21.15.69:55330) 2024-01-21T04:32:44.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.682+0000 7fc4effff640 1 -- 172.21.15.69:0/1052605214 learned_addr learned my addr 172.21.15.69:0/1052605214 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:32:44.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.682+0000 7fc4effff640 10 monclient(hunting): get_auth_request con 0x7fc4f01441e0 auth_method 0 2024-01-21T04:32:44.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.682+0000 7fc4effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:44.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.682+0000 7fc4effff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:44.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.682+0000 7fc4effff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:44.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.690+0000 7fc4effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:44.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.690+0000 7fc4effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:44.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.690+0000 7fc4effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:44.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.694+0000 7fc4f4afd640 1 --2- 172.21.15.69:0/1052605214 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4f0143bb0 0x7fc4f00ab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:44.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.701+0000 7fc4f52fe640 1 --2- 172.21.15.69:0/1052605214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4f01dada0 0x7fc4f01d6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:44.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.701+0000 7fc4effff640 10 monclient(hunting): handle_auth_done global_id 4107 payload 824 2024-01-21T04:32:44.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.701+0000 7fc4effff640 1 -- 172.21.15.69:0/1052605214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4f01dada0 msgr2=0x7fc4f01d6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:44.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.701+0000 7fc4effff640 1 --2- 172.21.15.69:0/1052605214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4f01dada0 0x7fc4f01d6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:44.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.701+0000 7fc4effff640 1 -- 172.21.15.69:0/1052605214 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4f0143bb0 msgr2=0x7fc4f00ab910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:44.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.701+0000 7fc4effff640 1 --2- 172.21.15.69:0/1052605214 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4f0143bb0 0x7fc4f00ab910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:44.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.701+0000 7fc4effff640 10 monclient: _finish_hunting 0 2024-01-21T04:32:44.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.701+0000 7fc4effff640 1 monclient: found mon.noname-a 2024-01-21T04:32:44.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.701+0000 7fc4effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:44.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.701+0000 7fc4effff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:44.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.701+0000 7fc4effff640 1 -- 172.21.15.69:0/1052605214 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f00acbe0 con 0x7fc4f01441e0 2024-01-21T04:32:44.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.704+0000 7fc4effff640 1 --2- 172.21.15.69:0/1052605214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f01441e0 0x7fc4f01da860 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009c00 tx=0x7fc4e400b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:44.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.704+0000 7fc4ed7fa640 10 client.?.objecter ms_handle_connect 0x7fc4f01441e0 2024-01-21T04:32:44.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.704+0000 7fc4ed7fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:44.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 1 -- 172.21.15.69:0/1052605214 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e4016660 con 0x7fc4f01441e0 2024-01-21T04:32:44.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:44.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:44.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 10 monclient: dump: 2024-01-21T04:32:44.719 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:32:44.719 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:44.719 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:44.719 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 10 monclient: _finish_auth 0 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 10 monclient: _check_auth_tickets 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 1 -- 172.21.15.69:0/1052605214 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e4011040 con 0x7fc4f01441e0 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4f6d61640 5 monclient: authenticate success, global_id 4107 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4f6d61640 10 monclient: _renew_subs 2024-01-21T04:32:44.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4f6d61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4f6d61640 1 -- 172.21.15.69:0/1052605214 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f0141110 con 0x7fc4f01441e0 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 1 -- 172.21.15.69:0/1052605214 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e40169e0 con 0x7fc4f01441e0 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4ed7fa640 10 monclient: dump: 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:44.721 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4eeffd640 4 set_mon_vals no callback set 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4f6d61640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4f6d61640 10 monclient: _renew_subs 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4f6d61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4f6d61640 1 -- 172.21.15.69:0/1052605214 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f01d74c0 con 0x7fc4f01441e0 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.717+0000 7fc4f6d61640 1 librados: init done 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.718+0000 7fc4f6d61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.718+0000 7fc4f6d61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.718+0000 7fc4f6d61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:44.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.718+0000 7fc4f6d61640 1 -- 172.21.15.69:0/1052605214 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f01da560 con 0x7fc4f01441e0 2024-01-21T04:32:44.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.902+0000 7fc4ed7fa640 1 -- 172.21.15.69:0/1052605214 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7fc4e4029430 con 0x7fc4f01441e0 2024-01-21T04:32:44.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.902+0000 7fc4ed7fa640 1 -- 172.21.15.69:0/1052605214 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc4e401f590 con 0x7fc4f01441e0 2024-01-21T04:32:44.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.902+0000 7fc4ed7fa640 10 client.4107.objecter ms_dispatch 0x7fc4f0000b90 osd_map(1..1 src has 1..1) v4 2024-01-21T04:32:44.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.902+0000 7fc4ed7fa640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-01-21T04:32:44.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.902+0000 7fc4ed7fa640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-01-21T04:32:44.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.902+0000 7fc4ed7fa640 20 client.4107.objecter dump_active .. 0 homeless 2024-01-21T04:32:44.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.902+0000 7fc4ed7fa640 1 -- 172.21.15.69:0/1052605214 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7fc4e401f890 con 0x7fc4f01441e0 2024-01-21T04:32:44.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.902+0000 7fc4ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:44.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:44.902+0000 7fc4ed7fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.008+0000 7fc4f6d61640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-21T04:32:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.008+0000 7fc4f6d61640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-21T04:32:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.008+0000 7fc4f6d61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.008+0000 7fc4f6d61640 1 -- 172.21.15.69:0/1052605214 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc4f004e970 con 0x7fc4f01441e0 2024-01-21T04:32:45.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.164+0000 7fc4ed7fa640 1 -- 172.21.15.69:0/1052605214 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc4e401fa30 con 0x7fc4f01441e0 2024-01-21T04:32:45.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.164+0000 7fc4ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-01-21T04:32:45.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.164+0000 7fc4ed7fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-01-21T04:32:45.166 INFO:teuthology.orchestra.run.smithi069.stderr:adjusted tunables profile to default 2024-01-21T04:32:45.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 10 librados: watch_flush enter 2024-01-21T04:32:45.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 10 librados: watch_flush exit 2024-01-21T04:32:45.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 20 client.4107.objecter shutdown clearing up homeless session... 2024-01-21T04:32:45.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 10 client.4107.objecter successfully canceled tick 2024-01-21T04:32:45.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 10 monclient: shutdown 2024-01-21T04:32:45.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:45.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 1 -- 172.21.15.69:0/1052605214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f01441e0 msgr2=0x7fc4f01da860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:45.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 1 --2- 172.21.15.69:0/1052605214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f01441e0 0x7fc4f01da860 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4009c00 tx=0x7fc4e400b800 comp rx=0 tx=0).stop 2024-01-21T04:32:45.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 1 -- 172.21.15.69:0/1052605214 shutdown_connections 2024-01-21T04:32:45.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 1 --2- 172.21.15.69:0/1052605214 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc4f01dada0 0x7fc4f01d6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:45.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 1 --2- 172.21.15.69:0/1052605214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4f01441e0 0x7fc4f01da860 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:45.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 1 --2- 172.21.15.69:0/1052605214 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4f0143bb0 0x7fc4f00ab910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:45.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 1 -- 172.21.15.69:0/1052605214 >> 172.21.15.69:0/1052605214 conn(0x7fc4f0066fb0 msgr2=0x7fc4f013e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:45.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 1 -- 172.21.15.69:0/1052605214 shutdown_connections 2024-01-21T04:32:45.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 1 -- 172.21.15.69:0/1052605214 wait complete. 2024-01-21T04:32:45.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:45.166+0000 7fc4f6d61640 1 librados: shutdown 2024-01-21T04:32:45.178 INFO:tasks.ceph:check_enable_crimson: False 2024-01-21T04:32:45.178 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-21T04:32:45.178 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:32:45.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-21T04:32:45.205 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:32:45.205 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-21T04:32:45.247 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:45.246+0000 7f4491032040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-21T04:32:45.273 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:32:45.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-21T04:32:45.303 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:45.302+0000 7f4491032040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-21T04:32:45.316 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:45.315+0000 7fc4fb42b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-21T04:32:45.329 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:32:45.329 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-21T04:32:45.369 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:45.368+0000 7fc4fb42b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-21T04:32:45.396 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:32:45.396 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-21T04:32:45.424 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:32:45.424 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-21T04:32:45.492 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:32:45.493 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-21T04:32:45.559 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-01-21T04:32:45.559 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-21T04:32:45.626 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 598f21db-ec64-41f8-b46f-6808b345bf82 0 2024-01-21T04:32:45.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.763+0000 7fa08f7dc640 10 monclient: get_monmap_and_config 2024-01-21T04:32:45.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.763+0000 7fa08f7dc640 10 monclient: build_initial_monmap 2024-01-21T04:32:45.764 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.763+0000 7fa08f7dc640 10 monclient: monmap: 2024-01-21T04:32:45.764 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:45.764 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:45.764 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:32:45.764207+0000 2024-01-21T04:32:45.764 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:32:45.764207+0000 2024-01-21T04:32:45.765 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:45.765 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:45.765 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:45.765 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:45.765 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:45.765 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:45.765 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:45.765 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:45.765 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.763+0000 7fa08f7dc640 1 Processor -- start 2024-01-21T04:32:45.765 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 1 -- start start 2024-01-21T04:32:45.765 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 10 monclient: init 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa0880a8b20 0x7fa088145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 10 monclient(hunting): picked mon.noname-a con 0x7fa0880a8b20 addr v2:172.21.15.69:3300/0 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880a9d10 0x7fa08813f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 10 monclient(hunting): picked mon.noname-b con 0x7fa0880a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0880b2310 addr v1:172.21.15.144:6789/0 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0881456f0 con 0x7fa0880b2310 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0880a98c0 con 0x7fa0880a8b20 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:45.767 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0881460c0 con 0x7fa0880a9d10 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.764+0000 7fa08f7dc640 10 monclient(hunting): authenticate will time out at 1288.080613s 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.765+0000 7fa08dd79640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fa0880b2310 0x7fa08813f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:40900/0 (socket says 172.21.15.144:40900) 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.765+0000 7fa08dd79640 1 -- 172.21.15.144:0/110849688 learned_addr learned my addr 172.21.15.144:0/110849688 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.765+0000 7fa08d578640 1 --2- 172.21.15.144:0/110849688 >> v2:172.21.15.69:3300/0 conn(0x7fa0880a8b20 0x7fa088145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.765+0000 7fa08cd77640 1 --2- 172.21.15.144:0/110849688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880a9d10 0x7fa08813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.765+0000 7fa08cd77640 10 monclient(hunting): get_auth_request con 0x7fa0880a9d10 auth_method 0 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.765+0000 7fa08cd77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.765+0000 7fa08cd77640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.765+0000 7fa08cd77640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.766+0000 7fa08d578640 1 -- 172.21.15.144:0/110849688 >> v2:172.21.15.69:3300/0 conn(0x7fa0880a8b20 msgr2=0x7fa088145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.766+0000 7fa08d578640 1 -- 172.21.15.144:0/110849688 >> v2:172.21.15.69:3300/0 conn(0x7fa0880a8b20 msgr2=0x7fa088145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.766+0000 7fa08d578640 1 --2- 172.21.15.144:0/110849688 >> v2:172.21.15.69:3300/0 conn(0x7fa0880a8b20 0x7fa088145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-21T04:32:45.768 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.766+0000 7fa08d578640 1 --2- 172.21.15.144:0/110849688 >> v2:172.21.15.69:3300/0 conn(0x7fa0880a8b20 0x7fa088145b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-21T04:32:45.769 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.766+0000 7fa08cd77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:45.769 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.766+0000 7fa08cd77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:45.769 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.766+0000 7fa08cd77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:45.769 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.767+0000 7fa08cd77640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-01-21T04:32:45.769 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.767+0000 7fa08cd77640 1 -- 172.21.15.144:0/110849688 >> v2:172.21.15.69:3300/0 conn(0x7fa0880a8b20 msgr2=0x7fa088145b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T04:32:45.769 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.767+0000 7fa08cd77640 1 --2- 172.21.15.144:0/110849688 >> v2:172.21.15.69:3300/0 conn(0x7fa0880a8b20 0x7fa088145b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:45.770 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.767+0000 7fa07ffff640 1 -- 172.21.15.144:0/110849688 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440250393 0 0) 0x7fa0881456f0 con 0x7fa0880b2310 2024-01-21T04:32:45.770 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.767+0000 7fa08cd77640 1 -- 172.21.15.144:0/110849688 >> v1:172.21.15.144:6789/0 conn(0x7fa0880b2310 legacy=0x7fa08813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:45.770 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.767+0000 7fa08cd77640 10 monclient: _finish_hunting 0 2024-01-21T04:32:45.770 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.767+0000 7fa08cd77640 1 monclient: found mon.noname-b 2024-01-21T04:32:45.770 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.767+0000 7fa08cd77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:45.770 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.767+0000 7fa08cd77640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:32:45.770 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.767+0000 7fa08cd77640 1 -- 172.21.15.144:0/110849688 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa088146230 con 0x7fa0880a9d10 2024-01-21T04:32:45.770 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.767+0000 7fa07ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-21T04:32:45.770 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.769+0000 7fa08cd77640 1 --2- 172.21.15.144:0/110849688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880a9d10 0x7fa08813f080 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa0780097a0 tx=0x7fa07802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d65052b30d920e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:45.775 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 1 -- 172.21.15.144:0/110849688 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07802e070 con 0x7fa0880a9d10 2024-01-21T04:32:45.775 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:45.775 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:32:45.775 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 10 monclient: dump: 2024-01-21T04:32:45.775 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:45.775 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:45.775 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 10 monclient: _finish_auth 0 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 10 monclient: _check_auth_tickets 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 1 -- 172.21.15.144:0/110849688 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0780315a0 con 0x7fa0880a9d10 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:45.776 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 1 -- 172.21.15.144:0/110849688 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa078031930 con 0x7fa0880a9d10 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa08f7dc640 5 monclient: authenticate success, global_id 4109 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa08f7dc640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa08f7dc640 4 set_mon_vals no callback set 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.774+0000 7fa07ffff640 10 monclient: dump: 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:45.777 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:45.778 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.775+0000 7fa08f7dc640 10 monclient: shutdown 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.775+0000 7fa08f7dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.775+0000 7fa08f7dc640 1 -- 172.21.15.144:0/110849688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880a9d10 msgr2=0x7fa08813f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.775+0000 7fa08f7dc640 1 --2- 172.21.15.144:0/110849688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880a9d10 0x7fa08813f080 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa0780097a0 tx=0x7fa07802b840 comp rx=0 tx=0).stop 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.775+0000 7fa08f7dc640 1 -- 172.21.15.144:0/110849688 shutdown_connections 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.775+0000 7fa08f7dc640 1 --2- 172.21.15.144:0/110849688 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880a9d10 0x7fa08813f080 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.775+0000 7fa08f7dc640 1 --2- 172.21.15.144:0/110849688 >> v2:172.21.15.69:3300/0 conn(0x7fa0880a8b20 0x7fa088145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.775+0000 7fa08f7dc640 1 -- 172.21.15.144:0/110849688 >> 172.21.15.144:0/110849688 conn(0x7fa088066fb0 msgr2=0x7fa0880b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.775+0000 7fa08f7dc640 1 -- 172.21.15.144:0/110849688 shutdown_connections 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.775+0000 7fa08f7dc640 1 -- 172.21.15.144:0/110849688 wait complete. 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.776+0000 7fa08f7dc640 10 monclient: build_initial_monmap 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.776+0000 7fa08f7dc640 10 monclient: monmap: 2024-01-21T04:32:45.781 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.776+0000 7fa08f7dc640 1 librados: starting msgr at 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.776+0000 7fa08f7dc640 1 librados: starting objecter 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.776+0000 7fa08f7dc640 1 Processor -- start 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.776+0000 7fa08f7dc640 1 -- start start 2024-01-21T04:32:45.782 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.776+0000 7fa08f7dc640 1 librados: setting wanted keys 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.776+0000 7fa08f7dc640 1 librados: calling monclient init 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.776+0000 7fa08f7dc640 10 monclient: init 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0880a8b20 0x7fa088142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 10 monclient(hunting): picked mon.noname-b con 0x7fa0880a8b20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0880a9d10 0x7fa088137f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 10 monclient(hunting): picked mon.noname-a con 0x7fa0880a9d10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880b2310 0x7fa0881d77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 10 monclient(hunting): picked mon.noname-c con 0x7fa0880b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa088141cd0 con 0x7fa0880a9d10 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0881d7280 con 0x7fa0880a8b20 2024-01-21T04:32:45.783 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08813fd60 con 0x7fa0880b2310 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.777+0000 7fa08f7dc640 10 monclient(hunting): authenticate will time out at 1288.093384s 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08d578640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0880a8b20 0x7fa088142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08cd77640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0880a9d10 0x7fa088137f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08dd79640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880b2310 0x7fa0881d77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08cd77640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0880a9d10 0x7fa088137f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.144:36900/0 (socket says 172.21.15.144:36900) 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08cd77640 1 -- 172.21.15.144:0/1466359765 learned_addr learned my addr 172.21.15.144:0/1466359765 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08cd77640 10 monclient(hunting): get_auth_request con 0x7fa0880a9d10 auth_method 0 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08cd77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08cd77640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08cd77640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08dd79640 10 monclient(hunting): get_auth_request con 0x7fa0880b2310 auth_method 0 2024-01-21T04:32:45.784 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08dd79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08dd79640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08dd79640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08d578640 10 monclient(hunting): get_auth_request con 0x7fa0880a8b20 auth_method 0 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08d578640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08d578640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08d578640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08dd79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08dd79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08dd79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08cd77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08cd77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.778+0000 7fa08cd77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.779+0000 7fa08d578640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.779+0000 7fa08d578640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:45.785 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.779+0000 7fa08d578640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:45.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.781+0000 7fa08dd79640 10 monclient(hunting): handle_auth_done global_id 4112 payload 824 2024-01-21T04:32:45.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.781+0000 7fa08dd79640 1 -- 172.21.15.144:0/1466359765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0880a8b20 msgr2=0x7fa088142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:45.786 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.781+0000 7fa08dd79640 1 --2- 172.21.15.144:0/1466359765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0880a8b20 0x7fa088142550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:45.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.781+0000 7fa08dd79640 1 -- 172.21.15.144:0/1466359765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0880a9d10 msgr2=0x7fa088137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:45.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.781+0000 7fa08dd79640 1 --2- 172.21.15.144:0/1466359765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0880a9d10 0x7fa088137f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:45.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.781+0000 7fa08dd79640 10 monclient: _finish_hunting 0 2024-01-21T04:32:45.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.781+0000 7fa08dd79640 1 monclient: found mon.noname-c 2024-01-21T04:32:45.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.781+0000 7fa08dd79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:45.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.781+0000 7fa08dd79640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:32:45.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.781+0000 7fa08dd79640 1 -- 172.21.15.144:0/1466359765 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa078009400 con 0x7fa0880b2310 2024-01-21T04:32:45.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.783+0000 7fa08dd79640 1 --2- 172.21.15.144:0/1466359765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880b2310 0x7fa0881d77a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa08000c750 tx=0x7fa08000cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:45.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.783+0000 7fa07dffb640 10 client.?.objecter ms_handle_connect 0x7fa0880b2310 2024-01-21T04:32:45.787 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.783+0000 7fa07dffb640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:45.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 1 -- 172.21.15.144:0/1466359765 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa080004d60 con 0x7fa0880b2310 2024-01-21T04:32:45.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:45.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:32:45.788 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 10 monclient: dump: 2024-01-21T04:32:45.788 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:45.788 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 10 monclient: _finish_auth 0 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 10 monclient: _check_auth_tickets 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 1 -- 172.21.15.144:0/1466359765 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08000a640 con 0x7fa0880b2310 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:45.789 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 1 -- 172.21.15.144:0/1466359765 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08000a9f0 con 0x7fa0880b2310 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07dffb640 10 monclient: dump: 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:45.790 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa08f7dc640 5 monclient: authenticate success, global_id 4112 2024-01-21T04:32:45.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa08f7dc640 10 monclient: _renew_subs 2024-01-21T04:32:45.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa08f7dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:45.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa08f7dc640 1 -- 172.21.15.144:0/1466359765 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa088138480 con 0x7fa0880b2310 2024-01-21T04:32:45.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa07effd640 4 set_mon_vals no callback set 2024-01-21T04:32:45.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa08f7dc640 10 client.4112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:45.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa08f7dc640 10 monclient: _renew_subs 2024-01-21T04:32:45.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa08f7dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:45.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa08f7dc640 1 -- 172.21.15.144:0/1466359765 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0881d7ec0 con 0x7fa0880b2310 2024-01-21T04:32:45.791 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.787+0000 7fa08f7dc640 1 librados: init done 2024-01-21T04:32:45.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.788+0000 7fa08f7dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:45.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.788+0000 7fa08f7dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:45.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.788+0000 7fa08f7dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:45.792 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.788+0000 7fa08f7dc640 1 -- 172.21.15.144:0/1466359765 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08804e970 con 0x7fa0880b2310 2024-01-21T04:32:45.843 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:45.842+0000 7f4491032040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-21T04:32:45.897 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:45.896+0000 7f4491032040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-21T04:32:45.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.904+0000 7fa07dffb640 1 -- 172.21.15.144:0/1466359765 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7fa0800070d0 con 0x7fa0880b2310 2024-01-21T04:32:45.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.904+0000 7fa07dffb640 1 -- 172.21.15.144:0/1466359765 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa0800125c0 con 0x7fa0880b2310 2024-01-21T04:32:45.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.904+0000 7fa07dffb640 10 client.4112.objecter ms_dispatch 0x7fa088000b90 osd_map(2..2 src has 1..2) v4 2024-01-21T04:32:45.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.904+0000 7fa07dffb640 3 client.4112.objecter handle_osd_map got epochs [2,2] > 0 2024-01-21T04:32:45.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.904+0000 7fa07dffb640 3 client.4112.objecter handle_osd_map decoding full epoch 2 2024-01-21T04:32:45.905 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:45.904+0000 7fa07dffb640 20 client.4112.objecter dump_active .. 0 homeless 2024-01-21T04:32:45.917 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:45.916+0000 7fc4fb42b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-21T04:32:45.943 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:45.942+0000 7f4491032040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-21T04:32:45.969 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:45.968+0000 7fc4fb42b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-21T04:32:45.998 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:45.998+0000 7f4491032040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-21T04:32:46.013 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:46.012+0000 7fc4fb42b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-21T04:32:46.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.029+0000 7fa07dffb640 1 -- 172.21.15.144:0/1466359765 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7fa080018070 con 0x7fa0880b2310 2024-01-21T04:32:46.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.029+0000 7fa07dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:46.030 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.029+0000 7fa07dffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:46.046 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:46.046+0000 7f4491032040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-21T04:32:46.067 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:46.066+0000 7fc4fb42b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-21T04:32:46.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.101+0000 7fa08f7dc640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "598f21db-ec64-41f8-b46f-6808b345bf82", "id": 0}] 2024-01-21T04:32:46.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.101+0000 7fa08f7dc640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "598f21db-ec64-41f8-b46f-6808b345bf82", "id": 0}] 2024-01-21T04:32:46.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.101+0000 7fa08f7dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:46.102 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.101+0000 7fa08f7dc640 1 -- 172.21.15.144:0/1466359765 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "598f21db-ec64-41f8-b46f-6808b345bf82", "id": 0} v 0) v1 -- 0x7fa0881d7440 con 0x7fa0880b2310 2024-01-21T04:32:46.110 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:46.109+0000 7fc4fb42b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-21T04:32:46.156 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:46.155+0000 7f4491032040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-21T04:32:46.211 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:46.209+0000 7fc4fb42b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-21T04:32:46.233 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:46.233+0000 7f4491032040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-21T04:32:46.282 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:46.281+0000 7fc4fb42b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-21T04:32:46.293 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:46.292+0000 7f4491032040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-21T04:32:46.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.321+0000 7fa07dffb640 1 -- 172.21.15.144:0/1466359765 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "598f21db-ec64-41f8-b46f-6808b345bf82", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa080007380 con 0x7fa0880b2310 2024-01-21T04:32:46.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.321+0000 7fa07dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "598f21db-ec64-41f8-b46f-6808b345bf82", "id": 0}] 2024-01-21T04:32:46.322 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.321+0000 7fa07dffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:32:46.323 INFO:teuthology.orchestra.run.smithi144.stdout:0 2024-01-21T04:32:46.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 10 librados: watch_flush enter 2024-01-21T04:32:46.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 10 librados: watch_flush exit 2024-01-21T04:32:46.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 20 client.4112.objecter shutdown clearing up homeless session... 2024-01-21T04:32:46.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 10 client.4112.objecter successfully canceled tick 2024-01-21T04:32:46.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 10 monclient: shutdown 2024-01-21T04:32:46.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:46.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 1 -- 172.21.15.144:0/1466359765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880b2310 msgr2=0x7fa0881d77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:46.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 1 --2- 172.21.15.144:0/1466359765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880b2310 0x7fa0881d77a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa08000c750 tx=0x7fa08000cc10 comp rx=0 tx=0).stop 2024-01-21T04:32:46.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 1 -- 172.21.15.144:0/1466359765 shutdown_connections 2024-01-21T04:32:46.324 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 1 --2- 172.21.15.144:0/1466359765 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa0880b2310 0x7fa0881d77a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 1 --2- 172.21.15.144:0/1466359765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0880a9d10 0x7fa088137f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 1 --2- 172.21.15.144:0/1466359765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa0880a8b20 0x7fa088142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 1 -- 172.21.15.144:0/1466359765 >> 172.21.15.144:0/1466359765 conn(0x7fa088066fb0 msgr2=0x7fa0880b5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:46.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 1 -- 172.21.15.144:0/1466359765 shutdown_connections 2024-01-21T04:32:46.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 1 -- 172.21.15.144:0/1466359765 wait complete. 2024-01-21T04:32:46.325 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.322+0000 7fa08f7dc640 1 librados: shutdown 2024-01-21T04:32:46.327 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:46.327+0000 7fc4fb42b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-21T04:32:46.333 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new f5228ed5-de32-48ef-9b87-809eab5bd25d 1 2024-01-21T04:32:46.345 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:46.345+0000 7f4491032040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-21T04:32:46.383 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:46.382+0000 7fc4fb42b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-21T04:32:46.392 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:46.391+0000 7f4491032040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-21T04:32:46.432 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:46.431+0000 7fc4fb42b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-21T04:32:46.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.432+0000 7fe3d2bc9640 10 monclient: get_monmap_and_config 2024-01-21T04:32:46.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.432+0000 7fe3d2bc9640 10 monclient: build_initial_monmap 2024-01-21T04:32:46.433 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.432+0000 7fe3d2bc9640 10 monclient: monmap: 2024-01-21T04:32:46.433 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:46.433 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:46.433 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:32:46.433494+0000 2024-01-21T04:32:46.433 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:32:46.433494+0000 2024-01-21T04:32:46.433 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:46.433 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.433 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:46.433 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:46.434 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:46.434 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:46.434 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:46.434 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.436 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 1 Processor -- start 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 1 -- start start 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 10 monclient: init 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe3cc0a8b20 0x7fe3cc145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 10 monclient(hunting): picked mon.noname-a con 0x7fe3cc0a8b20 addr v2:172.21.15.69:3300/0 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3cc0a9d10 0x7fe3cc13f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 10 monclient(hunting): picked mon.noname-b con 0x7fe3cc0a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3cc0b2310 addr v1:172.21.15.144:6789/0 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3cc1456f0 con 0x7fe3cc0b2310 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc0a98c0 con 0x7fe3cc0a8b20 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:46.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc1460c0 con 0x7fe3cc0a9d10 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.433+0000 7fe3d2bc9640 10 monclient(hunting): authenticate will time out at 1288.749856s 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.434+0000 7fe3d1166640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe3cc0b2310 0x7fe3cc13f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:40914/0 (socket says 172.21.15.144:40914) 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.434+0000 7fe3d1166640 1 -- 172.21.15.144:0/2378833541 learned_addr learned my addr 172.21.15.144:0/2378833541 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.434+0000 7fe3d0965640 1 --2- 172.21.15.144:0/2378833541 >> v2:172.21.15.69:3300/0 conn(0x7fe3cc0a8b20 0x7fe3cc145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.434+0000 7fe3cbfff640 1 --2- 172.21.15.144:0/2378833541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3cc0a9d10 0x7fe3cc13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.434+0000 7fe3d0965640 10 monclient(hunting): get_auth_request con 0x7fe3cc0a8b20 auth_method 0 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.434+0000 7fe3d0965640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.434+0000 7fe3d0965640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.434+0000 7fe3d0965640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3cbfff640 10 monclient(hunting): get_auth_request con 0x7fe3cc0a9d10 auth_method 0 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3cbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:46.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:46.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3d1166640 1 -- 172.21.15.144:0/2378833541 >> v1:172.21.15.144:6789/0 conn(0x7fe3cc0b2310 legacy=0x7fe3cc13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-01-21T04:32:46.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3d1166640 1 -- 172.21.15.144:0/2378833541 >> v1:172.21.15.144:6789/0 conn(0x7fe3cc0b2310 legacy=0x7fe3cc13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-21T04:32:46.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3d1166640 1 --1- 172.21.15.144:0/2378833541 >> v1:172.21.15.144:6789/0 conn(0x7fe3cc0b2310 0x7fe3cc13f5c0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-01-21T04:32:46.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3d0965640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:46.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3d0965640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:46.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3d0965640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:46.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:46.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:46.439 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.435+0000 7fe3cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:46.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.436+0000 7fe3d0965640 10 monclient(hunting): handle_auth_done global_id 4116 payload 274 2024-01-21T04:32:46.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.436+0000 7fe3d0965640 1 -- 172.21.15.144:0/2378833541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3cc0a9d10 msgr2=0x7fe3cc13f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:46.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.436+0000 7fe3d0965640 1 --2- 172.21.15.144:0/2378833541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3cc0a9d10 0x7fe3cc13f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.436+0000 7fe3d0965640 1 -- 172.21.15.144:0/2378833541 >> v1:172.21.15.144:6789/0 conn(0x7fe3cc0b2310 legacy=0x7fe3cc13f5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-21T04:32:46.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.436+0000 7fe3d0965640 10 monclient: _finish_hunting 0 2024-01-21T04:32:46.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.436+0000 7fe3d0965640 1 monclient: found mon.noname-a 2024-01-21T04:32:46.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.436+0000 7fe3d0965640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:46.440 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.436+0000 7fe3d0965640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.436+0000 7fe3d0965640 1 -- 172.21.15.144:0/2378833541 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc146230 con 0x7fe3cc0a8b20 2024-01-21T04:32:46.441 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.438+0000 7fe3d0965640 1 --2- 172.21.15.144:0/2378833541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3cc0a8b20 0x7fe3cc145b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00097a0 tx=0x7fe3c002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829815cf9321cdfb server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 1 -- 172.21.15.144:0/2378833541 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c002f050 con 0x7fe3cc0a8b20 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 10 monclient: dump: 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:46.443 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 10 monclient: _finish_auth 0 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 10 monclient: _check_auth_tickets 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 1 -- 172.21.15.144:0/2378833541 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c0031700 con 0x7fe3cc0a8b20 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3d2bc9640 5 monclient: authenticate success, global_id 4116 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3d2bc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3d2bc9640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3d2bc9640 4 set_mon_vals no callback set 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 1 -- 172.21.15.144:0/2378833541 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0031a90 con 0x7fe3cc0a8b20 2024-01-21T04:32:46.444 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3cb7fe640 10 monclient: dump: 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:46.445 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3d2bc9640 10 monclient: shutdown 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3d2bc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/2378833541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3cc0a8b20 msgr2=0x7fe3cc145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.442+0000 7fe3d2bc9640 1 --2- 172.21.15.144:0/2378833541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3cc0a8b20 0x7fe3cc145b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00097a0 tx=0x7fe3c002b7a0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.443+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/2378833541 shutdown_connections 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.443+0000 7fe3d2bc9640 1 --2- 172.21.15.144:0/2378833541 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3cc0a9d10 0x7fe3cc13f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.443+0000 7fe3d2bc9640 1 --2- 172.21.15.144:0/2378833541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3cc0a8b20 0x7fe3cc145b80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.443+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/2378833541 >> 172.21.15.144:0/2378833541 conn(0x7fe3cc066fb0 msgr2=0x7fe3cc0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.443+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/2378833541 shutdown_connections 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.443+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/2378833541 wait complete. 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.443+0000 7fe3d2bc9640 10 monclient: build_initial_monmap 2024-01-21T04:32:46.449 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.443+0000 7fe3d2bc9640 10 monclient: monmap: 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.443+0000 7fe3d2bc9640 1 librados: starting msgr at 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.443+0000 7fe3d2bc9640 1 librados: starting objecter 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 1 Processor -- start 2024-01-21T04:32:46.450 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 1 -- start start 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 1 librados: setting wanted keys 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 1 librados: calling monclient init 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 10 monclient: init 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3cc0a8b20 0x7fe3cc1368a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 10 monclient(hunting): picked mon.noname-c con 0x7fe3cc0a8b20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3cc0a9d10 0x7fe3cc13fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 10 monclient(hunting): picked mon.noname-b con 0x7fe3cc0a9d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3cc0b2310 0x7fe3cc141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 10 monclient(hunting): picked mon.noname-a con 0x7fe3cc0b2310 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc1421c0 con 0x7fe3cc0b2310 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:46.451 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc141fa0 con 0x7fe3cc0a9d10 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc138c80 con 0x7fe3cc0a8b20 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.444+0000 7fe3d2bc9640 10 monclient(hunting): authenticate will time out at 1288.760819s 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3cbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3cc0a9d10 0x7fe3cc13fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d1166640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3cc0b2310 0x7fe3cc141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d0965640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3cc0a8b20 0x7fe3cc1368a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d1166640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3cc0b2310 0x7fe3cc141a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.144:36916/0 (socket says 172.21.15.144:36916) 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d1166640 1 -- 172.21.15.144:0/1421687544 learned_addr learned my addr 172.21.15.144:0/1421687544 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d1166640 10 monclient(hunting): get_auth_request con 0x7fe3cc0b2310 auth_method 0 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d1166640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d1166640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:46.452 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d1166640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3cbfff640 10 monclient(hunting): get_auth_request con 0x7fe3cc0a9d10 auth_method 0 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3cbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d0965640 10 monclient(hunting): get_auth_request con 0x7fe3cc0a8b20 auth_method 0 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d0965640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d0965640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.445+0000 7fe3d0965640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.446+0000 7fe3d1166640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.446+0000 7fe3d1166640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.446+0000 7fe3d1166640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.446+0000 7fe3cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.446+0000 7fe3cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.446+0000 7fe3cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:46.453 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.446+0000 7fe3d0965640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:46.454 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.446+0000 7fe3d0965640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:46.454 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.446+0000 7fe3d0965640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:46.454 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.448+0000 7fe3d1166640 10 monclient(hunting): handle_auth_done global_id 4119 payload 824 2024-01-21T04:32:46.454 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.448+0000 7fe3d1166640 1 -- 172.21.15.144:0/1421687544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3cc0a8b20 msgr2=0x7fe3cc1368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.448+0000 7fe3d1166640 1 --2- 172.21.15.144:0/1421687544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3cc0a8b20 0x7fe3cc1368a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.448+0000 7fe3d1166640 1 -- 172.21.15.144:0/1421687544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3cc0a9d10 msgr2=0x7fe3cc13fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.448+0000 7fe3d1166640 1 --2- 172.21.15.144:0/1421687544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3cc0a9d10 0x7fe3cc13fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.448+0000 7fe3d1166640 10 monclient: _finish_hunting 0 2024-01-21T04:32:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.448+0000 7fe3d1166640 1 monclient: found mon.noname-a 2024-01-21T04:32:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.448+0000 7fe3d1166640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.448+0000 7fe3d1166640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.448+0000 7fe3d1166640 1 -- 172.21.15.144:0/1421687544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c0009400 con 0x7fe3cc0b2310 2024-01-21T04:32:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.451+0000 7fe3d1166640 1 --2- 172.21.15.144:0/1421687544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3cc0b2310 0x7fe3cc141a60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400cb20 tx=0x7fe3c4005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.451+0000 7fe3c97fa640 10 client.?.objecter ms_handle_connect 0x7fe3cc0b2310 2024-01-21T04:32:46.455 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.451+0000 7fe3c97fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3c97fa640 1 -- 172.21.15.144:0/1421687544 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400abd0 con 0x7fe3cc0b2310 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3c97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3c97fa640 10 monclient: dump: 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:46.457 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3c97fa640 10 monclient: _finish_auth 0 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3c97fa640 10 monclient: _check_auth_tickets 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3c97fa640 1 -- 172.21.15.144:0/1421687544 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c400ad30 con 0x7fe3cc0b2310 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3c97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3c97fa640 1 -- 172.21.15.144:0/1421687544 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400aea0 con 0x7fe3cc0b2310 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3d2bc9640 5 monclient: authenticate success, global_id 4119 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3d2bc9640 10 monclient: _renew_subs 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3d2bc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/1421687544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3cc138f60 con 0x7fe3cc0b2310 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.454+0000 7fe3caffd640 4 set_mon_vals no callback set 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:46.458 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3c97fa640 10 monclient: dump: 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3d2bc9640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3d2bc9640 10 monclient: _renew_subs 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3d2bc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.459 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/1421687544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc142930 con 0x7fe3cc0b2310 2024-01-21T04:32:46.460 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3d2bc9640 1 librados: init done 2024-01-21T04:32:46.460 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3d2bc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:46.460 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3d2bc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:46.460 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3d2bc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.460 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.455+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/1421687544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc04e970 con 0x7fe3cc0b2310 2024-01-21T04:32:46.552 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.551+0000 7fe3c97fa640 1 -- 172.21.15.144:0/1421687544 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7fe3c40129c0 con 0x7fe3cc0b2310 2024-01-21T04:32:46.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.556+0000 7fe3c97fa640 1 -- 172.21.15.144:0/1421687544 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fe3c401ce20 con 0x7fe3cc0b2310 2024-01-21T04:32:46.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.556+0000 7fe3c97fa640 10 client.4119.objecter ms_dispatch 0x7fe3cc000b90 osd_map(3..3 src has 1..3) v4 2024-01-21T04:32:46.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.556+0000 7fe3c97fa640 3 client.4119.objecter handle_osd_map got epochs [3,3] > 0 2024-01-21T04:32:46.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.556+0000 7fe3c97fa640 3 client.4119.objecter handle_osd_map decoding full epoch 3 2024-01-21T04:32:46.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.556+0000 7fe3c97fa640 20 client.4119.objecter dump_active .. 0 homeless 2024-01-21T04:32:46.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.556+0000 7fe3c97fa640 1 -- 172.21.15.144:0/1421687544 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7fe3c4017070 con 0x7fe3cc0b2310 2024-01-21T04:32:46.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.556+0000 7fe3c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:46.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.556+0000 7fe3c97fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:46.631 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.630+0000 7fe3d2bc9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f5228ed5-de32-48ef-9b87-809eab5bd25d", "id": 1}] 2024-01-21T04:32:46.631 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.630+0000 7fe3d2bc9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f5228ed5-de32-48ef-9b87-809eab5bd25d", "id": 1}] 2024-01-21T04:32:46.631 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.630+0000 7fe3d2bc9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.631 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.630+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/1421687544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f5228ed5-de32-48ef-9b87-809eab5bd25d", "id": 1} v 0) v1 -- 0x7fe3cc0b36a0 con 0x7fe3cc0b2310 2024-01-21T04:32:46.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.679+0000 7fe3c97fa640 1 -- 172.21.15.144:0/1421687544 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f5228ed5-de32-48ef-9b87-809eab5bd25d", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe3c401c510 con 0x7fe3cc0b2310 2024-01-21T04:32:46.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.679+0000 7fe3c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f5228ed5-de32-48ef-9b87-809eab5bd25d", "id": 1}] 2024-01-21T04:32:46.680 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.679+0000 7fe3c97fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:32:46.681 INFO:teuthology.orchestra.run.smithi144.stdout:1 2024-01-21T04:32:46.682 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.680+0000 7fe3d2bc9640 10 librados: watch_flush enter 2024-01-21T04:32:46.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 10 librados: watch_flush exit 2024-01-21T04:32:46.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 20 client.4119.objecter shutdown clearing up homeless session... 2024-01-21T04:32:46.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 10 client.4119.objecter successfully canceled tick 2024-01-21T04:32:46.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 10 monclient: shutdown 2024-01-21T04:32:46.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:46.683 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/1421687544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3cc0b2310 msgr2=0x7fe3cc141a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:46.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 1 --2- 172.21.15.144:0/1421687544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3cc0b2310 0x7fe3cc141a60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400cb20 tx=0x7fe3c4005b50 comp rx=0 tx=0).stop 2024-01-21T04:32:46.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/1421687544 shutdown_connections 2024-01-21T04:32:46.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 1 --2- 172.21.15.144:0/1421687544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3cc0b2310 0x7fe3cc141a60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 1 --2- 172.21.15.144:0/1421687544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3cc0a9d10 0x7fe3cc13fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 1 --2- 172.21.15.144:0/1421687544 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3cc0a8b20 0x7fe3cc1368a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/1421687544 >> 172.21.15.144:0/1421687544 conn(0x7fe3cc066fb0 msgr2=0x7fe3cc0b6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:46.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/1421687544 shutdown_connections 2024-01-21T04:32:46.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 1 -- 172.21.15.144:0/1421687544 wait complete. 2024-01-21T04:32:46.684 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.681+0000 7fe3d2bc9640 1 librados: shutdown 2024-01-21T04:32:46.694 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new b7482f36-5fda-454d-9088-40ab951e3120 2 2024-01-21T04:32:46.791 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:46.791+0000 7f4491032040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.792+0000 7f21af56c640 10 monclient: get_monmap_and_config 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.792+0000 7f21af56c640 10 monclient: build_initial_monmap 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.792+0000 7f21af56c640 10 monclient: monmap: 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:32:46.793247+0000 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:32:46.793247+0000 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:46.793 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:46.794 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:46.794 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:46.794 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.796 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.792+0000 7f21af56c640 1 Processor -- start 2024-01-21T04:32:46.796 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 1 -- start start 2024-01-21T04:32:46.796 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 10 monclient: init 2024-01-21T04:32:46.796 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:46.796 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f21a8070a10 0x7f21a8140380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 10 monclient(hunting): picked mon.noname-c con 0x7f21a8070a10 addr v2:172.21.15.144:3300/0 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21a8070df0 addr v1:172.21.15.144:6789/0 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f21a81408c0 0x7f21a8143b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 10 monclient(hunting): picked mon.noname-a con 0x7f21a81408c0 addr v2:172.21.15.69:3300/0 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a813fe80 con 0x7f21a8070df0 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a80a8610 con 0x7f21a81408c0 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a80a88f0 con 0x7f21a8070a10 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.793+0000 7f21af56c640 10 monclient(hunting): authenticate will time out at 1289.109662s 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21ad308640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f21a8070a10 0x7f21a8140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:46.797 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21adb09640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f21a81408c0 0x7f21a8143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21adb09640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f21a81408c0 0x7f21a8143b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.144:36928/0 (socket says 172.21.15.144:36928) 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21adb09640 1 -- 172.21.15.144:0/3891729591 learned_addr learned my addr 172.21.15.144:0/3891729591 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21adb09640 10 monclient(hunting): get_auth_request con 0x7f21a81408c0 auth_method 0 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21adb09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21adb09640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21adb09640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21ad308640 10 monclient(hunting): get_auth_request con 0x7f21a8070a10 auth_method 0 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21ad308640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21ad308640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.794+0000 7f21ad308640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.795+0000 7f21adb09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.795+0000 7f21adb09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.795+0000 7f21adb09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:46.798 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.795+0000 7f21ad308640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:46.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.795+0000 7f21ad308640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:46.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.795+0000 7f21ad308640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:46.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.797+0000 7f21adb09640 10 monclient(hunting): handle_auth_done global_id 4122 payload 274 2024-01-21T04:32:46.799 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.797+0000 7f21adb09640 1 -- 172.21.15.144:0/3891729591 >> v2:172.21.15.144:3300/0 conn(0x7f21a8070a10 msgr2=0x7f21a8140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:46.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.797+0000 7f21adb09640 1 --2- 172.21.15.144:0/3891729591 >> v2:172.21.15.144:3300/0 conn(0x7f21a8070a10 0x7f21a8140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.797+0000 7f21adb09640 1 -- 172.21.15.144:0/3891729591 >> v1:172.21.15.144:6789/0 conn(0x7f21a8070df0 legacy=0x7f21a8143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:46.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.797+0000 7f21adb09640 10 monclient: _finish_hunting 0 2024-01-21T04:32:46.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.797+0000 7f21adb09640 1 monclient: found mon.noname-a 2024-01-21T04:32:46.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.797+0000 7f21adb09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:46.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.797+0000 7f21adb09640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.797+0000 7f21adb09640 1 -- 172.21.15.144:0/3891729591 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a80a8bd0 con 0x7f21a81408c0 2024-01-21T04:32:46.800 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.799+0000 7f21adb09640 1 --2- 172.21.15.144:0/3891729591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21a81408c0 0x7f21a8143b10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f21a8070730 tx=0x7f21a002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbbe5f6aafe3c80 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:46.801 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.801+0000 7f219ffff640 1 -- 172.21.15.144:0/3891729591 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a002cb20 con 0x7f21a81408c0 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.801+0000 7f219ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.801+0000 7f219ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.801+0000 7f219ffff640 10 monclient: dump: 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.802 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.801+0000 7f219ffff640 10 monclient: _finish_auth 0 2024-01-21T04:32:46.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.801+0000 7f219ffff640 10 monclient: _check_auth_tickets 2024-01-21T04:32:46.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.801+0000 7f219ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:46.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.801+0000 7f21af56c640 5 monclient: authenticate success, global_id 4122 2024-01-21T04:32:46.803 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.801+0000 7f21af56c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:32:46.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.802+0000 7f219ffff640 1 -- 172.21.15.144:0/3891729591 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a002cc80 con 0x7f21a81408c0 2024-01-21T04:32:46.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.802+0000 7f219ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:46.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.802+0000 7f21af56c640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:46.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.802+0000 7f219ffff640 1 -- 172.21.15.144:0/3891729591 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a00343f0 con 0x7f21a81408c0 2024-01-21T04:32:46.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.802+0000 7f21af56c640 4 set_mon_vals no callback set 2024-01-21T04:32:46.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.802+0000 7f219ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:46.804 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.802+0000 7f219ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.802+0000 7f219ffff640 10 monclient: dump: 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.803+0000 7f21af56c640 10 monclient: shutdown 2024-01-21T04:32:46.805 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.803+0000 7f21af56c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:46.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.803+0000 7f21af56c640 1 -- 172.21.15.144:0/3891729591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21a81408c0 msgr2=0x7f21a8143b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:46.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.803+0000 7f21af56c640 1 --2- 172.21.15.144:0/3891729591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21a81408c0 0x7f21a8143b10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f21a8070730 tx=0x7f21a002bf80 comp rx=0 tx=0).stop 2024-01-21T04:32:46.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.803+0000 7f21af56c640 1 -- 172.21.15.144:0/3891729591 shutdown_connections 2024-01-21T04:32:46.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.803+0000 7f21af56c640 1 --2- 172.21.15.144:0/3891729591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21a81408c0 0x7f21a8143b10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.803+0000 7f21af56c640 1 --2- 172.21.15.144:0/3891729591 >> v2:172.21.15.144:3300/0 conn(0x7f21a8070a10 0x7f21a8140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.803+0000 7f21af56c640 1 -- 172.21.15.144:0/3891729591 >> 172.21.15.144:0/3891729591 conn(0x7f21a8066f50 msgr2=0x7f21a80b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:46.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.803+0000 7f21af56c640 1 -- 172.21.15.144:0/3891729591 shutdown_connections 2024-01-21T04:32:46.806 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.803+0000 7f21af56c640 1 -- 172.21.15.144:0/3891729591 wait complete. 2024-01-21T04:32:46.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.804+0000 7f21af56c640 10 monclient: build_initial_monmap 2024-01-21T04:32:46.810 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.804+0000 7f21af56c640 10 monclient: monmap: 2024-01-21T04:32:46.810 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:46.810 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:46.810 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-01-21T04:32:46.810 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-01-21T04:32:46.810 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:46.810 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.810 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:46.811 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:46.811 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:46.811 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.804+0000 7f21af56c640 1 librados: starting msgr at 2024-01-21T04:32:46.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.804+0000 7f21af56c640 1 librados: starting objecter 2024-01-21T04:32:46.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.804+0000 7f21af56c640 1 Processor -- start 2024-01-21T04:32:46.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.804+0000 7f21af56c640 1 -- start start 2024-01-21T04:32:46.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.804+0000 7f21af56c640 1 librados: setting wanted keys 2024-01-21T04:32:46.811 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.804+0000 7f21af56c640 1 librados: calling monclient init 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.804+0000 7f21af56c640 10 monclient: init 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21a8070a10 0x7f21a80ad950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 10 monclient(hunting): picked mon.noname-a con 0x7f21a8070a10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21a8070df0 0x7f21a80acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 10 monclient(hunting): picked mon.noname-c con 0x7f21a8070df0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21a81408c0 0x7f21a81d5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 10 monclient(hunting): picked mon.noname-b con 0x7f21a81408c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a81d5140 con 0x7f21a8070a10 2024-01-21T04:32:46.812 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80aefa0 con 0x7f21a81408c0 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80a9f30 con 0x7f21a8070df0 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21af56c640 10 monclient(hunting): authenticate will time out at 1289.121257s 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21adb09640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21a81408c0 0x7f21a81d5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21ad308640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21a8070a10 0x7f21a80ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.805+0000 7f21acb07640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21a8070df0 0x7f21a80acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21ad308640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21a8070a10 0x7f21a80ad950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.144:36940/0 (socket says 172.21.15.144:36940) 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21ad308640 1 -- 172.21.15.144:0/3325824898 learned_addr learned my addr 172.21.15.144:0/3325824898 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21ad308640 10 monclient(hunting): get_auth_request con 0x7f21a8070a10 auth_method 0 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21ad308640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21ad308640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:46.813 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21ad308640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21adb09640 10 monclient(hunting): get_auth_request con 0x7f21a81408c0 auth_method 0 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21adb09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21adb09640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21adb09640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21acb07640 10 monclient(hunting): get_auth_request con 0x7f21a8070df0 auth_method 0 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21acb07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21acb07640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21acb07640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21ad308640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21ad308640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21ad308640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21adb09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21adb09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:46.814 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21adb09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21acb07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21acb07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.806+0000 7f21acb07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.809+0000 7f21ad308640 10 monclient(hunting): handle_auth_done global_id 4125 payload 824 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.809+0000 7f21ad308640 1 -- 172.21.15.144:0/3325824898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21a8070df0 msgr2=0x7f21a80acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.809+0000 7f21ad308640 1 --2- 172.21.15.144:0/3325824898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21a8070df0 0x7f21a80acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.809+0000 7f21ad308640 1 -- 172.21.15.144:0/3325824898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21a81408c0 msgr2=0x7f21a81d5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.809+0000 7f21ad308640 1 --2- 172.21.15.144:0/3325824898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21a81408c0 0x7f21a81d5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.809+0000 7f21ad308640 10 monclient: _finish_hunting 0 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.809+0000 7f21ad308640 1 monclient: found mon.noname-a 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.809+0000 7f21ad308640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.809+0000 7f21ad308640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.815 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.809+0000 7f21ad308640 1 -- 172.21.15.144:0/3325824898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0009b30 con 0x7f21a8070a10 2024-01-21T04:32:46.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.811+0000 7f21ad308640 1 --2- 172.21.15.144:0/3325824898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21a8070a10 0x7f21a80ad950 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f21a400e9b0 tx=0x7f21a400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:46.817 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.811+0000 7f219dffb640 10 client.?.objecter ms_handle_connect 0x7f21a8070a10 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.811+0000 7f219dffb640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.813+0000 7f219dffb640 1 -- 172.21.15.144:0/3325824898 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a4009670 con 0x7f21a8070a10 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.813+0000 7f219dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.813+0000 7f219dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.813+0000 7f219dffb640 10 monclient: dump: 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:46.818 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.813+0000 7f219dffb640 10 monclient: _finish_auth 0 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.813+0000 7f219dffb640 10 monclient: _check_auth_tickets 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.813+0000 7f219dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.813+0000 7f21af56c640 5 monclient: authenticate success, global_id 4125 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.813+0000 7f21af56c640 10 monclient: _renew_subs 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.813+0000 7f21af56c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.813+0000 7f21af56c640 1 -- 172.21.15.144:0/3325824898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a80ad0f0 con 0x7f21a8070a10 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f21af56c640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f21af56c640 10 monclient: _renew_subs 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f21af56c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f21af56c640 1 -- 172.21.15.144:0/3325824898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a81d5c80 con 0x7f21a8070a10 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f21af56c640 1 librados: init done 2024-01-21T04:32:46.819 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f219dffb640 1 -- 172.21.15.144:0/3325824898 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a400c9a0 con 0x7f21a8070a10 2024-01-21T04:32:46.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f219dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:46.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f219dffb640 1 -- 172.21.15.144:0/3325824898 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a400cd10 con 0x7f21a8070a10 2024-01-21T04:32:46.820 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f219dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:46.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f219dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:46.823 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f219dffb640 10 monclient: dump: 2024-01-21T04:32:46.824 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:46.824 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:46.824 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.824 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:46.824 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:46.824 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:46.824 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:46.824 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:46.824 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:46.825 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:46.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.814+0000 7f219f7fe640 4 set_mon_vals no callback set 2024-01-21T04:32:46.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.815+0000 7f21af56c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:46.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.815+0000 7f21af56c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:46.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.815+0000 7f21af56c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.815+0000 7f21af56c640 1 -- 172.21.15.144:0/3325824898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2188005080 con 0x7f21a8070a10 2024-01-21T04:32:46.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.817+0000 7f219dffb640 1 -- 172.21.15.144:0/3325824898 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f21a4005990 con 0x7f21a8070a10 2024-01-21T04:32:46.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.817+0000 7f219dffb640 1 -- 172.21.15.144:0/3325824898 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f21a4013890 con 0x7f21a8070a10 2024-01-21T04:32:46.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.817+0000 7f219dffb640 10 client.4125.objecter ms_dispatch 0x7f21a8000b90 osd_map(4..4 src has 1..4) v4 2024-01-21T04:32:46.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.817+0000 7f219dffb640 3 client.4125.objecter handle_osd_map got epochs [4,4] > 0 2024-01-21T04:32:46.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.817+0000 7f219dffb640 3 client.4125.objecter handle_osd_map decoding full epoch 4 2024-01-21T04:32:46.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.817+0000 7f219dffb640 20 client.4125.objecter dump_active .. 0 homeless 2024-01-21T04:32:46.832 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:46.832+0000 7fc4fb42b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-21T04:32:46.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.915+0000 7f219dffb640 1 -- 172.21.15.144:0/3325824898 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f21a4021060 con 0x7f21a8070a10 2024-01-21T04:32:46.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.915+0000 7f219dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:46.916 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.915+0000 7f219dffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:46.930 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:46.929+0000 7f4491032040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-21T04:32:46.968 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:46.967+0000 7fc4fb42b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-21T04:32:46.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.987+0000 7f21af56c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b7482f36-5fda-454d-9088-40ab951e3120", "id": 2}] 2024-01-21T04:32:46.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.987+0000 7f21af56c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b7482f36-5fda-454d-9088-40ab951e3120", "id": 2}] 2024-01-21T04:32:46.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.987+0000 7f21af56c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:46.988 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:46.987+0000 7f21af56c640 1 -- 172.21.15.144:0/3325824898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b7482f36-5fda-454d-9088-40ab951e3120", "id": 2} v 0) v1 -- 0x7f21880056a0 con 0x7f21a8070a10 2024-01-21T04:32:47.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.023+0000 7f219dffb640 1 -- 172.21.15.144:0/3325824898 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b7482f36-5fda-454d-9088-40ab951e3120", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f21a4017070 con 0x7f21a8070a10 2024-01-21T04:32:47.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.023+0000 7f219dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b7482f36-5fda-454d-9088-40ab951e3120", "id": 2}] 2024-01-21T04:32:47.024 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.023+0000 7f219dffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:32:47.024 INFO:teuthology.orchestra.run.smithi144.stdout:2 2024-01-21T04:32:47.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.024+0000 7f21af56c640 10 librados: watch_flush enter 2024-01-21T04:32:47.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 10 librados: watch_flush exit 2024-01-21T04:32:47.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 20 client.4125.objecter shutdown clearing up homeless session... 2024-01-21T04:32:47.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 10 client.4125.objecter successfully canceled tick 2024-01-21T04:32:47.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 10 monclient: shutdown 2024-01-21T04:32:47.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:47.026 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 1 -- 172.21.15.144:0/3325824898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21a8070a10 msgr2=0x7f21a80ad950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 1 --2- 172.21.15.144:0/3325824898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21a8070a10 0x7f21a80ad950 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f21a400e9b0 tx=0x7f21a400ee70 comp rx=0 tx=0).stop 2024-01-21T04:32:47.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 1 -- 172.21.15.144:0/3325824898 shutdown_connections 2024-01-21T04:32:47.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 1 --2- 172.21.15.144:0/3325824898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f21a81408c0 0x7f21a81d5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 1 --2- 172.21.15.144:0/3325824898 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f21a8070df0 0x7f21a80acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 1 --2- 172.21.15.144:0/3325824898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21a8070a10 0x7f21a80ad950 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 1 -- 172.21.15.144:0/3325824898 >> 172.21.15.144:0/3325824898 conn(0x7f21a8066f50 msgr2=0x7f21a8143750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:47.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 1 -- 172.21.15.144:0/3325824898 shutdown_connections 2024-01-21T04:32:47.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 1 -- 172.21.15.144:0/3325824898 wait complete. 2024-01-21T04:32:47.027 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.025+0000 7f21af56c640 1 librados: shutdown 2024-01-21T04:32:47.039 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new 429b547e-67bc-4d00-8f97-84d9cc4ad69a 3 2024-01-21T04:32:47.088 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:47.087+0000 7fc4fb42b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-21T04:32:47.136 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:47.135+0000 7fc4fb42b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-21T04:32:47.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.138+0000 7fe20e32c640 10 monclient: get_monmap_and_config 2024-01-21T04:32:47.139 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.138+0000 7fe20e32c640 10 monclient: build_initial_monmap 2024-01-21T04:32:47.140 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.138+0000 7fe20e32c640 10 monclient: monmap: 2024-01-21T04:32:47.140 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:47.140 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.140 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:32:47.139645+0000 2024-01-21T04:32:47.140 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:32:47.139645+0000 2024-01-21T04:32:47.140 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:47.141 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.141 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:47.141 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:47.141 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:47.141 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:47.141 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:47.141 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.139+0000 7fe20e32c640 1 Processor -- start 2024-01-21T04:32:47.143 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.139+0000 7fe20e32c640 1 -- start start 2024-01-21T04:32:47.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.139+0000 7fe20e32c640 10 monclient: init 2024-01-21T04:32:47.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.139+0000 7fe20e32c640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:47.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.139+0000 7fe20e32c640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-01-21T04:32:47.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe20e32c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2080a8b20 addr v1:172.21.15.144:6789/0 2024-01-21T04:32:47.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe20e32c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2080a9d10 0x7fe20813f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe20e32c640 10 monclient(hunting): picked mon.noname-a con 0x7fe2080a9d10 addr v2:172.21.15.69:3300/0 2024-01-21T04:32:47.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe20e32c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2080b2310 addr v1:172.21.15.69:6789/0 2024-01-21T04:32:47.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe20e32c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2081456f0 con 0x7fe2080b2310 2024-01-21T04:32:47.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe20e32c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2080a98c0 con 0x7fe2080a8b20 2024-01-21T04:32:47.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe20e32c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.144 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe20e32c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe208146290 con 0x7fe2080a9d10 2024-01-21T04:32:47.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe20e32c640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:47.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe20e32c640 10 monclient(hunting): authenticate will time out at 1289.455991s 2024-01-21T04:32:47.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe207fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe2080a8b20 0x7fe208145b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:40930/0 (socket says 172.21.15.144:40930) 2024-01-21T04:32:47.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.140+0000 7fe207fff640 1 -- 172.21.15.144:0/2461617659 learned_addr learned my addr 172.21.15.144:0/2461617659 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-01-21T04:32:47.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.141+0000 7fe2077fe640 1 --2- 172.21.15.144:0/2461617659 >> v2:172.21.15.69:3300/0 conn(0x7fe2080a9d10 0x7fe20813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.141+0000 7fe2077fe640 10 monclient(hunting): get_auth_request con 0x7fe2080a9d10 auth_method 0 2024-01-21T04:32:47.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.141+0000 7fe2077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.141+0000 7fe2077fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.141+0000 7fe2077fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.141+0000 7fe207fff640 1 -- 172.21.15.144:0/2461617659 >> v1:172.21.15.144:6789/0 conn(0x7fe2080a8b20 legacy=0x7fe208145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-01-21T04:32:47.145 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.141+0000 7fe207fff640 1 -- 172.21.15.144:0/2461617659 >> v1:172.21.15.144:6789/0 conn(0x7fe2080a8b20 legacy=0x7fe208145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-21T04:32:47.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.141+0000 7fe207fff640 1 --1- 172.21.15.144:0/2461617659 >> v1:172.21.15.144:6789/0 conn(0x7fe2080a8b20 0x7fe208145b80 :-1 s=OPENED pgs=19 cs=1 l=1).handle_message read tag failed 2024-01-21T04:32:47.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.141+0000 7fe207fff640 1 --1- 172.21.15.144:0/2461617659 >> v1:172.21.15.144:6789/0 conn(0x7fe2080a8b20 0x7fe208145b80 :-1 s=OPENED pgs=19 cs=1 l=1).fault on lossy channel, failing 2024-01-21T04:32:47.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.141+0000 7fe206ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.144:6789/0 2024-01-21T04:32:47.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.142+0000 7fe2077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.142+0000 7fe2077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.146 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.142+0000 7fe2077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.143+0000 7fe2077fe640 10 monclient(hunting): handle_auth_done global_id 4128 payload 274 2024-01-21T04:32:47.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.143+0000 7fe2077fe640 1 -- 172.21.15.144:0/2461617659 >> v1:172.21.15.144:6789/0 conn(0x7fe2080a8b20 legacy=0x7fe208145b80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-01-21T04:32:47.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.143+0000 7fe2077fe640 1 -- 172.21.15.144:0/2461617659 >> v1:172.21.15.69:6789/0 conn(0x7fe2080b2310 legacy=0x7fe20813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.143+0000 7fe2077fe640 10 monclient: _finish_hunting 0 2024-01-21T04:32:47.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.143+0000 7fe2077fe640 1 monclient: found mon.noname-a 2024-01-21T04:32:47.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.143+0000 7fe2077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:47.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.143+0000 7fe2077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.147 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.143+0000 7fe2077fe640 1 -- 172.21.15.144:0/2461617659 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2081463d0 con 0x7fe2080a9d10 2024-01-21T04:32:47.152 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.151+0000 7fe2077fe640 1 --2- 172.21.15.144:0/2461617659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2080a9d10 0x7fe20813f080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe2080aa660 tx=0x7fe1f802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d8a3426e2ba4b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:47.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 1 -- 172.21.15.144:0/2461617659 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8031800 con 0x7fe2080a9d10 2024-01-21T04:32:47.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:47.156 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 10 monclient: dump: 2024-01-21T04:32:47.156 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 10 monclient: _finish_auth 0 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 10 monclient: _check_auth_tickets 2024-01-21T04:32:47.157 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 1 -- 172.21.15.144:0/2461617659 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f8031960 con 0x7fe2080a9d10 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe20e32c640 5 monclient: authenticate success, global_id 4128 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe20e32c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe20e32c640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe20e32c640 4 set_mon_vals no callback set 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 1 -- 172.21.15.144:0/2461617659 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8031cf0 con 0x7fe2080a9d10 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.155+0000 7fe206ffd640 10 monclient: dump: 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.158 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.159 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.159 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.159 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.159 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.159 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.159 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.159 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.159 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.163 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.156+0000 7fe20e32c640 10 monclient: shutdown 2024-01-21T04:32:47.163 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.156+0000 7fe20e32c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:47.163 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.156+0000 7fe20e32c640 1 -- 172.21.15.144:0/2461617659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2080a9d10 msgr2=0x7fe20813f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.156+0000 7fe20e32c640 1 --2- 172.21.15.144:0/2461617659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2080a9d10 0x7fe20813f080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe2080aa660 tx=0x7fe1f802b8a0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.156+0000 7fe20e32c640 1 -- 172.21.15.144:0/2461617659 shutdown_connections 2024-01-21T04:32:47.164 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.156+0000 7fe20e32c640 1 --2- 172.21.15.144:0/2461617659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2080a9d10 0x7fe20813f080 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.156+0000 7fe20e32c640 1 -- 172.21.15.144:0/2461617659 >> 172.21.15.144:0/2461617659 conn(0x7fe208066fb0 msgr2=0x7fe2080b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:47.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.156+0000 7fe20e32c640 1 -- 172.21.15.144:0/2461617659 shutdown_connections 2024-01-21T04:32:47.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.156+0000 7fe20e32c640 1 -- 172.21.15.144:0/2461617659 wait complete. 2024-01-21T04:32:47.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.156+0000 7fe20e32c640 10 monclient: build_initial_monmap 2024-01-21T04:32:47.165 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.157+0000 7fe20e32c640 10 monclient: monmap: 2024-01-21T04:32:47.165 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:47.165 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:47.165 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-01-21T04:32:47.165 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-01-21T04:32:47.165 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:47.165 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.166 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:47.166 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:47.166 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:47.166 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.157+0000 7fe20e32c640 1 librados: starting msgr at 2024-01-21T04:32:47.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.157+0000 7fe20e32c640 1 librados: starting objecter 2024-01-21T04:32:47.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.157+0000 7fe20e32c640 1 Processor -- start 2024-01-21T04:32:47.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.157+0000 7fe20e32c640 1 -- start start 2024-01-21T04:32:47.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.157+0000 7fe20e32c640 1 librados: setting wanted keys 2024-01-21T04:32:47.166 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.157+0000 7fe20e32c640 1 librados: calling monclient init 2024-01-21T04:32:47.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.157+0000 7fe20e32c640 10 monclient: init 2024-01-21T04:32:47.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:47.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T04:32:47.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2080a8b20 0x7fe208136ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 10 monclient(hunting): picked mon.noname-b con 0x7fe2080a8b20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:47.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2080a9d10 0x7fe208138d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 10 monclient(hunting): picked mon.noname-a con 0x7fe2080a9d10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:47.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2080b2310 0x7fe2081319f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 10 monclient(hunting): picked mon.noname-c con 0x7fe2080b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:47.167 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2081374a0 con 0x7fe2080a9d10 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe208137930 con 0x7fe2080a8b20 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe208138680 con 0x7fe2080b2310 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20e32c640 10 monclient(hunting): authenticate will time out at 1289.474097s 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe207fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2080a8b20 0x7fe208136ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe2077fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2080a9d10 0x7fe208138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe2077fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2080a9d10 0x7fe208138d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.144:36966/0 (socket says 172.21.15.144:36966) 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe2077fe640 1 -- 172.21.15.144:0/2325429447 learned_addr learned my addr 172.21.15.144:0/2325429447 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-01-21T04:32:47.168 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe20c8c9640 1 --2- 172.21.15.144:0/2325429447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2080b2310 0x7fe2081319f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe2077fe640 10 monclient(hunting): get_auth_request con 0x7fe2080a9d10 auth_method 0 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe2077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe2077fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.158+0000 7fe2077fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe207fff640 10 monclient(hunting): get_auth_request con 0x7fe2080a8b20 auth_method 0 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe207fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe207fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe207fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe20c8c9640 10 monclient(hunting): get_auth_request con 0x7fe2080b2310 auth_method 0 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe20c8c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe20c8c9640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.169 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe20c8c9640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe2077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe2077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe2077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe207fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe207fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe207fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe20c8c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe20c8c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.159+0000 7fe20c8c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.162+0000 7fe2077fe640 10 monclient(hunting): handle_auth_done global_id 4134 payload 824 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.162+0000 7fe2077fe640 1 -- 172.21.15.144:0/2325429447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2080b2310 msgr2=0x7fe2081319f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.170 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.162+0000 7fe2077fe640 1 --2- 172.21.15.144:0/2325429447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2080b2310 0x7fe2081319f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.162+0000 7fe2077fe640 1 -- 172.21.15.144:0/2325429447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2080a8b20 msgr2=0x7fe208136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.162+0000 7fe2077fe640 1 --2- 172.21.15.144:0/2325429447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2080a8b20 0x7fe208136ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.162+0000 7fe2077fe640 10 monclient: _finish_hunting 0 2024-01-21T04:32:47.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.162+0000 7fe2077fe640 1 monclient: found mon.noname-a 2024-01-21T04:32:47.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.162+0000 7fe2077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:47.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.162+0000 7fe2077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.171 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.162+0000 7fe2077fe640 1 -- 172.21.15.144:0/2325429447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f8009400 con 0x7fe2080a9d10 2024-01-21T04:32:47.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.164+0000 7fe2077fe640 1 --2- 172.21.15.144:0/2325429447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2080a9d10 0x7fe208138d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe1f802beb0 tx=0x7fe1f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:47.172 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.164+0000 7fe204ff9640 10 client.?.objecter ms_handle_connect 0x7fe2080a9d10 2024-01-21T04:32:47.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.164+0000 7fe204ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:47.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.167+0000 7fe204ff9640 1 -- 172.21.15.144:0/2325429447 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f803be70 con 0x7fe2080a9d10 2024-01-21T04:32:47.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.167+0000 7fe204ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.167+0000 7fe204ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:47.173 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.167+0000 7fe204ff9640 10 monclient: dump: 2024-01-21T04:32:47.173 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.173 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.173 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.173 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.173 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.173 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.167+0000 7fe204ff9640 10 monclient: _finish_auth 0 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.167+0000 7fe204ff9640 10 monclient: _check_auth_tickets 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.167+0000 7fe204ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.167+0000 7fe204ff9640 1 -- 172.21.15.144:0/2325429447 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f80453f0 con 0x7fe2080a9d10 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.167+0000 7fe204ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.167+0000 7fe204ff9640 1 -- 172.21.15.144:0/2325429447 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f80457a0 con 0x7fe2080a9d10 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.167+0000 7fe204ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.174 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe204ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe204ff9640 10 monclient: dump: 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 5 monclient: authenticate success, global_id 4134 2024-01-21T04:32:47.175 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 10 monclient: _renew_subs 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe2067fc640 4 set_mon_vals no callback set 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 1 -- 172.21.15.144:0/2325429447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe208138960 con 0x7fe2080a9d10 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 10 monclient: _renew_subs 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 1 -- 172.21.15.144:0/2325429447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe208142920 con 0x7fe2080a9d10 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 1 librados: init done 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.176 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.168+0000 7fe20e32c640 1 -- 172.21.15.144:0/2325429447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2080b5dd0 con 0x7fe2080a9d10 2024-01-21T04:32:47.235 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:47.234+0000 7fc4fb42b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-21T04:32:47.262 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.261+0000 7fe204ff9640 1 -- 172.21.15.144:0/2325429447 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7fe1f8045900 con 0x7fe2080a9d10 2024-01-21T04:32:47.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.265+0000 7fe204ff9640 1 -- 172.21.15.144:0/2325429447 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fe1f802e070 con 0x7fe2080a9d10 2024-01-21T04:32:47.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.265+0000 7fe204ff9640 10 client.4134.objecter ms_dispatch 0x7fe208000b90 osd_map(5..5 src has 1..5) v4 2024-01-21T04:32:47.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.265+0000 7fe204ff9640 3 client.4134.objecter handle_osd_map got epochs [5,5] > 0 2024-01-21T04:32:47.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.265+0000 7fe204ff9640 3 client.4134.objecter handle_osd_map decoding full epoch 5 2024-01-21T04:32:47.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.265+0000 7fe204ff9640 20 client.4134.objecter dump_active .. 0 homeless 2024-01-21T04:32:47.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.265+0000 7fe204ff9640 1 -- 172.21.15.144:0/2325429447 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7fe1f8053060 con 0x7fe2080a9d10 2024-01-21T04:32:47.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.265+0000 7fe204ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:47.266 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.265+0000 7fe204ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:47.331 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:47.330+0000 7fc4fb42b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-21T04:32:47.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.337+0000 7fe20e32c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "429b547e-67bc-4d00-8f97-84d9cc4ad69a", "id": 3}] 2024-01-21T04:32:47.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.337+0000 7fe20e32c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "429b547e-67bc-4d00-8f97-84d9cc4ad69a", "id": 3}] 2024-01-21T04:32:47.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.337+0000 7fe20e32c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.338 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.337+0000 7fe20e32c640 1 -- 172.21.15.144:0/2325429447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "429b547e-67bc-4d00-8f97-84d9cc4ad69a", "id": 3} v 0) v1 -- 0x7fe20804e970 con 0x7fe2080a9d10 2024-01-21T04:32:47.359 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:47.358+0000 7f4491032040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-21T04:32:47.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.393+0000 7fe204ff9640 1 -- 172.21.15.144:0/2325429447 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "429b547e-67bc-4d00-8f97-84d9cc4ad69a", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe1f8035020 con 0x7fe2080a9d10 2024-01-21T04:32:47.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.393+0000 7fe204ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "429b547e-67bc-4d00-8f97-84d9cc4ad69a", "id": 3}] 2024-01-21T04:32:47.394 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.393+0000 7fe204ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:32:47.395 INFO:teuthology.orchestra.run.smithi144.stdout:3 2024-01-21T04:32:47.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 10 librados: watch_flush enter 2024-01-21T04:32:47.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 10 librados: watch_flush exit 2024-01-21T04:32:47.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 20 client.4134.objecter shutdown clearing up homeless session... 2024-01-21T04:32:47.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 10 client.4134.objecter successfully canceled tick 2024-01-21T04:32:47.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 10 monclient: shutdown 2024-01-21T04:32:47.396 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:47.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 1 -- 172.21.15.144:0/2325429447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2080a9d10 msgr2=0x7fe208138d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 1 --2- 172.21.15.144:0/2325429447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2080a9d10 0x7fe208138d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe1f802beb0 tx=0x7fe1f80046c0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 1 -- 172.21.15.144:0/2325429447 shutdown_connections 2024-01-21T04:32:47.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 1 --2- 172.21.15.144:0/2325429447 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe2080b2310 0x7fe2081319f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 1 --2- 172.21.15.144:0/2325429447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2080a9d10 0x7fe208138d80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 1 --2- 172.21.15.144:0/2325429447 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2080a8b20 0x7fe208136ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 1 -- 172.21.15.144:0/2325429447 >> 172.21.15.144:0/2325429447 conn(0x7fe208066fb0 msgr2=0x7fe2080b31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:47.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 1 -- 172.21.15.144:0/2325429447 shutdown_connections 2024-01-21T04:32:47.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 1 -- 172.21.15.144:0/2325429447 wait complete. 2024-01-21T04:32:47.397 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.394+0000 7fe20e32c640 1 librados: shutdown 2024-01-21T04:32:47.406 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new ca5cd910-9006-47e5-a83d-1d2533c24871 4 2024-01-21T04:32:47.409 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:47.408+0000 7f4491032040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-21T04:32:47.479 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:47.478+0000 7fc4fb42b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-21T04:32:47.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.503+0000 7f464e875640 10 monclient: get_monmap_and_config 2024-01-21T04:32:47.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.503+0000 7f464e875640 10 monclient: build_initial_monmap 2024-01-21T04:32:47.504 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.503+0000 7f464e875640 10 monclient: monmap: 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:32:47.504627+0000 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:32:47.504627+0000 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:47.505 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.504+0000 7f464e875640 1 Processor -- start 2024-01-21T04:32:47.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.504+0000 7f464e875640 1 -- start start 2024-01-21T04:32:47.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.504+0000 7f464e875640 10 monclient: init 2024-01-21T04:32:47.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:47.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-01-21T04:32:47.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f46480a8b20 0x7f4648145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 10 monclient(hunting): picked mon.noname-a con 0x7f46480a8b20 addr v2:172.21.15.69:3300/0 2024-01-21T04:32:47.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f46480a9d10 0x7f464813f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 10 monclient(hunting): picked mon.noname-c con 0x7f46480a9d10 addr v2:172.21.15.144:3300/0 2024-01-21T04:32:47.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46480b2310 addr v1:172.21.15.69:6789/0 2024-01-21T04:32:47.508 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46481456f0 con 0x7f46480b2310 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46480a98c0 con 0x7f46480a8b20 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46481460c0 con 0x7f46480a9d10 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464e875640 10 monclient(hunting): authenticate will time out at 1289.821068s 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f46477fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f46480a9d10 0x7f464813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f4647fff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f46480a8b20 0x7f4648145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464ce12640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f46480b2310 0x7f464813f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.144:60020/0 (socket says 172.21.15.144:60020) 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.505+0000 7f464ce12640 1 -- 172.21.15.144:0/1125468357 learned_addr learned my addr 172.21.15.144:0/1125468357 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f4647fff640 10 monclient(hunting): get_auth_request con 0x7f46480a8b20 auth_method 0 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f4647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f4647fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.509 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f4647fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f46477fe640 10 monclient(hunting): get_auth_request con 0x7f46480a9d10 auth_method 0 2024-01-21T04:32:47.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f46477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f46477fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f46477fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f46477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f46477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f46477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f4647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.506+0000 7f4647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.510 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.507+0000 7f4647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.508+0000 7f4647fff640 10 monclient(hunting): handle_auth_done global_id 4137 payload 274 2024-01-21T04:32:47.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.508+0000 7f4647fff640 1 -- 172.21.15.144:0/1125468357 >> v2:172.21.15.144:3300/0 conn(0x7f46480a9d10 msgr2=0x7f464813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:47.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.508+0000 7f4647fff640 1 --2- 172.21.15.144:0/1125468357 >> v2:172.21.15.144:3300/0 conn(0x7f46480a9d10 0x7f464813f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.508+0000 7f4647fff640 1 -- 172.21.15.144:0/1125468357 >> v1:172.21.15.69:6789/0 conn(0x7f46480b2310 legacy=0x7f464813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.508+0000 7f4647fff640 10 monclient: _finish_hunting 0 2024-01-21T04:32:47.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.508+0000 7f4647fff640 1 monclient: found mon.noname-a 2024-01-21T04:32:47.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.508+0000 7f4647fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:47.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.508+0000 7f4647fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.511 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.508+0000 7f4647fff640 1 -- 172.21.15.144:0/1125468357 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4648146230 con 0x7f46480a8b20 2024-01-21T04:32:47.512 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.511+0000 7f4647fff640 1 --2- 172.21.15.144:0/1125468357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480a8b20 0x7f4648145b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f46480aa660 tx=0x7f463802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e77ef040e3905e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:47.512 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:47.511+0000 7f4491032040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-21T04:32:47.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 1 -- 172.21.15.144:0/1125468357 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638031830 con 0x7f46480a8b20 2024-01-21T04:32:47.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:47.518 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 10 monclient: dump: 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 10 monclient: _finish_auth 0 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 10 monclient: _check_auth_tickets 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 1 -- 172.21.15.144:0/1125468357 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4638031990 con 0x7f46480a8b20 2024-01-21T04:32:47.519 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 1 -- 172.21.15.144:0/1125468357 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638031d20 con 0x7f46480a8b20 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f464e875640 5 monclient: authenticate success, global_id 4137 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f464e875640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f464e875640 4 set_mon_vals no callback set 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.517+0000 7f4646ffd640 10 monclient: dump: 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.520 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.521 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.521 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.521 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.521 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.518+0000 7f464e875640 10 monclient: shutdown 2024-01-21T04:32:47.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.518+0000 7f464e875640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:47.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.518+0000 7f464e875640 1 -- 172.21.15.144:0/1125468357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480a8b20 msgr2=0x7f4648145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.518+0000 7f464e875640 1 --2- 172.21.15.144:0/1125468357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480a8b20 0x7f4648145b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f46480aa660 tx=0x7f463802b8d0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.518+0000 7f464e875640 1 -- 172.21.15.144:0/1125468357 shutdown_connections 2024-01-21T04:32:47.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.518+0000 7f464e875640 1 --2- 172.21.15.144:0/1125468357 >> v2:172.21.15.144:3300/0 conn(0x7f46480a9d10 0x7f464813f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.518+0000 7f464e875640 1 --2- 172.21.15.144:0/1125468357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480a8b20 0x7f4648145b80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.518+0000 7f464e875640 1 -- 172.21.15.144:0/1125468357 >> 172.21.15.144:0/1125468357 conn(0x7f4648066fb0 msgr2=0x7f46480b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:47.524 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.518+0000 7f464e875640 1 -- 172.21.15.144:0/1125468357 shutdown_connections 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.518+0000 7f464e875640 1 -- 172.21.15.144:0/1125468357 wait complete. 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.519+0000 7f464e875640 10 monclient: build_initial_monmap 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.519+0000 7f464e875640 10 monclient: monmap: 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:47.525 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.519+0000 7f464e875640 1 librados: starting msgr at 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.519+0000 7f464e875640 1 librados: starting objecter 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.519+0000 7f464e875640 1 Processor -- start 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.519+0000 7f464e875640 1 -- start start 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.519+0000 7f464e875640 1 librados: setting wanted keys 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.519+0000 7f464e875640 1 librados: calling monclient init 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.519+0000 7f464e875640 10 monclient: init 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480a8b20 0x7f46481da870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 10 monclient(hunting): picked mon.noname-a con 0x7f46480a8b20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46480a9d10 0x7f4648140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.526 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 10 monclient(hunting): picked mon.noname-b con 0x7f46480a9d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46480b2310 0x7f46481daeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 10 monclient(hunting): picked mon.noname-c con 0x7f46480b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46481405d0 con 0x7f46480a8b20 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4648137dd0 con 0x7f46480a9d10 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4648139ae0 con 0x7f46480b2310 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f464e875640 10 monclient(hunting): authenticate will time out at 1289.836428s 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.520+0000 7f4647fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480a8b20 0x7f46481da870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f46477fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46480a9d10 0x7f4648140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f464ce12640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46480b2310 0x7f46481daeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.527 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f4647fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480a8b20 0x7f46481da870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.144:36978/0 (socket says 172.21.15.144:36978) 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f4647fff640 1 -- 172.21.15.144:0/2051376153 learned_addr learned my addr 172.21.15.144:0/2051376153 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f4647fff640 10 monclient(hunting): get_auth_request con 0x7f46480a8b20 auth_method 0 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f4647fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f4647fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f4647fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f46477fe640 10 monclient(hunting): get_auth_request con 0x7f46480a9d10 auth_method 0 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f46477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f46477fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f46477fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f464ce12640 10 monclient(hunting): get_auth_request con 0x7f46480b2310 auth_method 0 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f464ce12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f464ce12640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f464ce12640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.528 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f4647fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f4647fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f4647fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f46477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f46477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.521+0000 7f46477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.522+0000 7f464ce12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.522+0000 7f464ce12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.529 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.522+0000 7f464ce12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.529 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:47.523+0000 7fc4fb42b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-21T04:32:47.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.524+0000 7f4647fff640 10 monclient(hunting): handle_auth_done global_id 4143 payload 824 2024-01-21T04:32:47.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.524+0000 7f4647fff640 1 -- 172.21.15.144:0/2051376153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46480b2310 msgr2=0x7f46481daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.524+0000 7f4647fff640 1 --2- 172.21.15.144:0/2051376153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46480b2310 0x7f46481daeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.524+0000 7f4647fff640 1 -- 172.21.15.144:0/2051376153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46480a9d10 msgr2=0x7f4648140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.524+0000 7f4647fff640 1 --2- 172.21.15.144:0/2051376153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46480a9d10 0x7f4648140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.524+0000 7f4647fff640 10 monclient: _finish_hunting 0 2024-01-21T04:32:47.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.524+0000 7f4647fff640 1 monclient: found mon.noname-a 2024-01-21T04:32:47.531 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.524+0000 7f4647fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.524+0000 7f4647fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.524+0000 7f4647fff640 1 -- 172.21.15.144:0/2051376153 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4638009400 con 0x7f46480a8b20 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.526+0000 7f4647fff640 1 --2- 172.21.15.144:0/2051376153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480a8b20 0x7f46481da870 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4638009890 tx=0x7f46380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.526+0000 7f4644ff9640 10 client.?.objecter ms_handle_connect 0x7f46480a8b20 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.526+0000 7f4644ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 1 -- 172.21.15.144:0/2051376153 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463803b970 con 0x7f46480a8b20 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 10 monclient: dump: 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.532 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 10 monclient: _finish_auth 0 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 10 monclient: _check_auth_tickets 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 1 -- 172.21.15.144:0/2051376153 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463803bad0 con 0x7f46480a8b20 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f464e875640 5 monclient: authenticate success, global_id 4143 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f464e875640 10 monclient: _renew_subs 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f464e875640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f464e875640 1 -- 172.21.15.144:0/2051376153 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4648138070 con 0x7f46480a8b20 2024-01-21T04:32:47.533 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 1 -- 172.21.15.144:0/2051376153 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463803be80 con 0x7f46480a8b20 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.529+0000 7f4644ff9640 10 monclient: dump: 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.534 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.530+0000 7f464e875640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:47.535 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.530+0000 7f464e875640 10 monclient: _renew_subs 2024-01-21T04:32:47.535 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.530+0000 7f464e875640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.535 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.530+0000 7f46467fc640 4 set_mon_vals no callback set 2024-01-21T04:32:47.535 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.530+0000 7f464e875640 1 -- 172.21.15.144:0/2051376153 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4648142f90 con 0x7f46480a8b20 2024-01-21T04:32:47.535 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.530+0000 7f464e875640 1 librados: init done 2024-01-21T04:32:47.536 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.531+0000 7f464e875640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:47.536 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.531+0000 7f464e875640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:47.536 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.531+0000 7f464e875640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.536 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.531+0000 7f464e875640 1 -- 172.21.15.144:0/2051376153 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464804e970 con 0x7f46480a8b20 2024-01-21T04:32:47.536 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.534+0000 7f4644ff9640 1 -- 172.21.15.144:0/2051376153 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f463804e770 con 0x7f46480a8b20 2024-01-21T04:32:47.536 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.534+0000 7f4644ff9640 1 -- 172.21.15.144:0/2051376153 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f46380443a0 con 0x7f46480a8b20 2024-01-21T04:32:47.536 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.534+0000 7f4644ff9640 10 client.4143.objecter ms_dispatch 0x7f4648000b90 osd_map(6..6 src has 1..6) v4 2024-01-21T04:32:47.536 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.534+0000 7f4644ff9640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-01-21T04:32:47.536 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.534+0000 7f4644ff9640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-01-21T04:32:47.536 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.535+0000 7f4644ff9640 20 client.4143.objecter dump_active .. 0 homeless 2024-01-21T04:32:47.612 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:47.611+0000 7f4491032040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-21T04:32:47.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.636+0000 7f4644ff9640 1 -- 172.21.15.144:0/2051376153 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f4638053060 con 0x7f46480a8b20 2024-01-21T04:32:47.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.636+0000 7f4644ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:47.640 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.636+0000 7f4644ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:47.668 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:47.668+0000 7fc4fb42b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-21T04:32:47.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.708+0000 7f464e875640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ca5cd910-9006-47e5-a83d-1d2533c24871", "id": 4}] 2024-01-21T04:32:47.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.708+0000 7f464e875640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ca5cd910-9006-47e5-a83d-1d2533c24871", "id": 4}] 2024-01-21T04:32:47.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.708+0000 7f464e875640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.709 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.708+0000 7f464e875640 1 -- 172.21.15.144:0/2051376153 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ca5cd910-9006-47e5-a83d-1d2533c24871", "id": 4} v 0) v1 -- 0x7f4648138200 con 0x7f46480a8b20 2024-01-21T04:32:47.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.739+0000 7f4644ff9640 1 -- 172.21.15.144:0/2051376153 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ca5cd910-9006-47e5-a83d-1d2533c24871", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4638035020 con 0x7f46480a8b20 2024-01-21T04:32:47.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.739+0000 7f4644ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ca5cd910-9006-47e5-a83d-1d2533c24871", "id": 4}] 2024-01-21T04:32:47.740 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.739+0000 7f4644ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:32:47.740 INFO:teuthology.orchestra.run.smithi144.stdout:4 2024-01-21T04:32:47.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 10 librados: watch_flush enter 2024-01-21T04:32:47.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 10 librados: watch_flush exit 2024-01-21T04:32:47.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 20 client.4143.objecter shutdown clearing up homeless session... 2024-01-21T04:32:47.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 10 client.4143.objecter successfully canceled tick 2024-01-21T04:32:47.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 10 monclient: shutdown 2024-01-21T04:32:47.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:47.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 1 -- 172.21.15.144:0/2051376153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480a8b20 msgr2=0x7f46481da870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 1 --2- 172.21.15.144:0/2051376153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480a8b20 0x7f46481da870 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4638009890 tx=0x7f46380046c0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 1 -- 172.21.15.144:0/2051376153 shutdown_connections 2024-01-21T04:32:47.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 1 --2- 172.21.15.144:0/2051376153 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f46480b2310 0x7f46481daeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.742 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 1 --2- 172.21.15.144:0/2051376153 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46480a9d10 0x7f4648140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 1 --2- 172.21.15.144:0/2051376153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46480a8b20 0x7f46481da870 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 1 -- 172.21.15.144:0/2051376153 >> 172.21.15.144:0/2051376153 conn(0x7f4648066fb0 msgr2=0x7f464813ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:47.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 1 -- 172.21.15.144:0/2051376153 shutdown_connections 2024-01-21T04:32:47.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 1 -- 172.21.15.144:0/2051376153 wait complete. 2024-01-21T04:32:47.743 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.740+0000 7f464e875640 1 librados: shutdown 2024-01-21T04:32:47.752 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new e10f1b39-2112-44cf-8045-8b3bc316d07b 5 2024-01-21T04:32:47.758 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:47.758+0000 7f4491032040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-21T04:32:47.802 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:47.801+0000 7f4491032040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-21T04:32:47.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.850+0000 7f11151a9640 10 monclient: get_monmap_and_config 2024-01-21T04:32:47.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.850+0000 7f11151a9640 10 monclient: build_initial_monmap 2024-01-21T04:32:47.851 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.850+0000 7f11151a9640 10 monclient: monmap: 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:32:47.851599+0000 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:32:47.851599+0000 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.852 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 1 Processor -- start 2024-01-21T04:32:47.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 1 -- start start 2024-01-21T04:32:47.853 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 10 monclient: init 2024-01-21T04:32:47.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:47.854 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f11100a8b20 0x7f1110145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 10 monclient(hunting): picked mon.noname-c con 0x7f11100a8b20 addr v2:172.21.15.144:3300/0 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11100a9d10 addr v1:172.21.15.69:6789/0 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11100b2310 0x7f111013f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 10 monclient(hunting): picked mon.noname-b con 0x7f11100b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11101456f0 con 0x7f11100a9d10 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11100a98c0 con 0x7f11100a8b20 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11101460c0 con 0x7f11100b2310 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.851+0000 7f11151a9640 10 monclient(hunting): authenticate will time out at 1290.167799s 2024-01-21T04:32:47.855 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.852+0000 7f110ed9d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f11100a8b20 0x7f1110145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.852+0000 7f110e59c640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f11100a9d10 0x7f111013f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.144:60030/0 (socket says 172.21.15.144:60030) 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.852+0000 7f110e59c640 1 -- 172.21.15.144:0/623712814 learned_addr learned my addr 172.21.15.144:0/623712814 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.852+0000 7f110f59e640 1 --2- 172.21.15.144:0/623712814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11100b2310 0x7f111013f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.852+0000 7f110ed9d640 10 monclient(hunting): get_auth_request con 0x7f11100a8b20 auth_method 0 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.852+0000 7f110ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.852+0000 7f110ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.852+0000 7f110ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.853+0000 7f110f59e640 10 monclient(hunting): get_auth_request con 0x7f11100b2310 auth_method 0 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.853+0000 7f110f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.853+0000 7f110f59e640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.853+0000 7f110f59e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.853+0000 7f110ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.856 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.853+0000 7f110ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.853+0000 7f110ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.853+0000 7f110f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.853+0000 7f110f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.853+0000 7f110f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.857 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.854+0000 7f110ed9d640 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2024-01-21T04:32:47.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.854+0000 7f110ed9d640 1 -- 172.21.15.144:0/623712814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11100b2310 msgr2=0x7f111013f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:47.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.854+0000 7f110ed9d640 1 --2- 172.21.15.144:0/623712814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11100b2310 0x7f111013f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.854+0000 7f110ed9d640 1 -- 172.21.15.144:0/623712814 >> v1:172.21.15.69:6789/0 conn(0x7f11100a9d10 legacy=0x7f111013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.854+0000 7f110ed9d640 10 monclient: _finish_hunting 0 2024-01-21T04:32:47.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.854+0000 7f110ed9d640 1 monclient: found mon.noname-c 2024-01-21T04:32:47.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.854+0000 7f110ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:47.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.854+0000 7f110ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:32:47.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.854+0000 7f110ed9d640 1 -- 172.21.15.144:0/623712814 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1110146230 con 0x7f11100a8b20 2024-01-21T04:32:47.858 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.857+0000 7f110ed9d640 1 --2- 172.21.15.144:0/623712814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11100a8b20 0x7f1110145b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f11000097a0 tx=0x7f110002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9cca674ce587689 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:47.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 1 -- 172.21.15.144:0/623712814 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110002e070 con 0x7f11100a8b20 2024-01-21T04:32:47.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:32:47.862 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 10 monclient: dump: 2024-01-21T04:32:47.862 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.862 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.862 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 10 monclient: _finish_auth 0 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 10 monclient: _check_auth_tickets 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 1 -- 172.21.15.144:0/623712814 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11000315a0 con 0x7f11100a8b20 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 1 -- 172.21.15.144:0/623712814 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100031930 con 0x7f11100a8b20 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:32:47.863 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f110dd9b640 10 monclient: dump: 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f11151a9640 5 monclient: authenticate success, global_id 4141 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f11151a9640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:47.864 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.861+0000 7f11151a9640 4 set_mon_vals no callback set 2024-01-21T04:32:47.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.862+0000 7f11151a9640 10 monclient: shutdown 2024-01-21T04:32:47.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.862+0000 7f11151a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:47.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.862+0000 7f11151a9640 1 -- 172.21.15.144:0/623712814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11100a8b20 msgr2=0x7f1110145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.862+0000 7f11151a9640 1 --2- 172.21.15.144:0/623712814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11100a8b20 0x7f1110145b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f11000097a0 tx=0x7f110002b840 comp rx=0 tx=0).stop 2024-01-21T04:32:47.867 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.862+0000 7f11151a9640 1 -- 172.21.15.144:0/623712814 shutdown_connections 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.862+0000 7f11151a9640 1 --2- 172.21.15.144:0/623712814 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11100b2310 0x7f111013f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.862+0000 7f11151a9640 1 --2- 172.21.15.144:0/623712814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11100a8b20 0x7f1110145b80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.862+0000 7f11151a9640 1 -- 172.21.15.144:0/623712814 >> 172.21.15.144:0/623712814 conn(0x7f1110066fb0 msgr2=0x7f11100b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.862+0000 7f11151a9640 1 -- 172.21.15.144:0/623712814 shutdown_connections 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.862+0000 7f11151a9640 1 -- 172.21.15.144:0/623712814 wait complete. 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.863+0000 7f11151a9640 10 monclient: build_initial_monmap 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.863+0000 7f11151a9640 10 monclient: monmap: 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:47.868 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.863+0000 7f11151a9640 1 librados: starting msgr at 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.863+0000 7f11151a9640 1 librados: starting objecter 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 1 Processor -- start 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 1 -- start start 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 1 librados: setting wanted keys 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 1 librados: calling monclient init 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 10 monclient: init 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11100a8b20 0x7f1110136590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 10 monclient(hunting): picked mon.noname-a con 0x7f11100a8b20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:47.869 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11100a9d10 0x7f1110142080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 10 monclient(hunting): picked mon.noname-b con 0x7f11100a9d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11100b2310 0x7f1110141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 10 monclient(hunting): picked mon.noname-c con 0x7f11100b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1110142750 con 0x7f11100a8b20 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1110142be0 con 0x7f11100a9d10 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11101329e0 con 0x7f11100b2310 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.864+0000 7f11151a9640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f11151a9640 10 monclient(hunting): authenticate will time out at 1290.180937s 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110e59c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11100a9d10 0x7f1110142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11100a8b20 0x7f1110136590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.870 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110f59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11100b2310 0x7f1110141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11100a8b20 0x7f1110136590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.144:36992/0 (socket says 172.21.15.144:36992) 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110ed9d640 1 -- 172.21.15.144:0/2627634392 learned_addr learned my addr 172.21.15.144:0/2627634392 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110ed9d640 10 monclient(hunting): get_auth_request con 0x7f11100a8b20 auth_method 0 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110e59c640 10 monclient(hunting): get_auth_request con 0x7f11100a9d10 auth_method 0 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110e59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110f59e640 10 monclient(hunting): get_auth_request con 0x7f11100b2310 auth_method 0 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110f59e640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.865+0000 7f110f59e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.866+0000 7f110ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.871 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.866+0000 7f110ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.866+0000 7f110ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.866+0000 7f110e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.866+0000 7f110e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.866+0000 7f110e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.866+0000 7f110f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:47.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.866+0000 7f110f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:47.872 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.866+0000 7f110f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:47.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.868+0000 7f110ed9d640 10 monclient(hunting): handle_auth_done global_id 4149 payload 824 2024-01-21T04:32:47.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.868+0000 7f110ed9d640 1 -- 172.21.15.144:0/2627634392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11100b2310 msgr2=0x7f1110141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.868+0000 7f110ed9d640 1 --2- 172.21.15.144:0/2627634392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11100b2310 0x7f1110141570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.868+0000 7f110ed9d640 1 -- 172.21.15.144:0/2627634392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11100a9d10 msgr2=0x7f1110142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:47.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.868+0000 7f110ed9d640 1 --2- 172.21.15.144:0/2627634392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11100a9d10 0x7f1110142080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:47.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.868+0000 7f110ed9d640 10 monclient: _finish_hunting 0 2024-01-21T04:32:47.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.868+0000 7f110ed9d640 1 monclient: found mon.noname-a 2024-01-21T04:32:47.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.868+0000 7f110ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:47.873 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.868+0000 7f110ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.868+0000 7f110ed9d640 1 -- 172.21.15.144:0/2627634392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1100009400 con 0x7f11100a8b20 2024-01-21T04:32:47.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.871+0000 7f110ed9d640 1 --2- 172.21.15.144:0/2627634392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11100a8b20 0x7f1110136590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1100034040 tx=0x7f11000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:47.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.871+0000 7f10f77fe640 10 client.?.objecter ms_handle_connect 0x7f11100a8b20 2024-01-21T04:32:47.874 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.871+0000 7f10f77fe640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 1 -- 172.21.15.144:0/2627634392 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100032840 con 0x7f11100a8b20 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 10 monclient: dump: 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.876 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 10 monclient: _finish_auth 0 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 10 monclient: _check_auth_tickets 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 1 -- 172.21.15.144:0/2627634392 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11000329c0 con 0x7f11100a8b20 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 1 -- 172.21.15.144:0/2627634392 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100032d70 con 0x7f11100a8b20 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f10f77fe640 10 monclient: dump: 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:47.877 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f11151a9640 5 monclient: authenticate success, global_id 4149 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f11151a9640 10 monclient: _renew_subs 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f11151a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f11151a9640 1 -- 172.21.15.144:0/2627634392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1110132c00 con 0x7f11100a8b20 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f110d59a640 4 set_mon_vals no callback set 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f11151a9640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f11151a9640 10 monclient: _renew_subs 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f11151a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f11151a9640 1 -- 172.21.15.144:0/2627634392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11101330c0 con 0x7f11100a8b20 2024-01-21T04:32:47.878 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.874+0000 7f11151a9640 1 librados: init done 2024-01-21T04:32:47.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.875+0000 7f11151a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:47.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.875+0000 7f11151a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:47.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.875+0000 7f11151a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:47.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.875+0000 7f11151a9640 1 -- 172.21.15.144:0/2627634392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111004e970 con 0x7f11100a8b20 2024-01-21T04:32:47.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.877+0000 7f10f77fe640 1 -- 172.21.15.144:0/2627634392 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f1100005d10 con 0x7f11100a8b20 2024-01-21T04:32:47.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.877+0000 7f10f77fe640 1 -- 172.21.15.144:0/2627634392 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f110004eaa0 con 0x7f11100a8b20 2024-01-21T04:32:47.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.877+0000 7f10f77fe640 10 client.4149.objecter ms_dispatch 0x7f1110000b90 osd_map(7..7 src has 1..7) v4 2024-01-21T04:32:47.879 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.877+0000 7f10f77fe640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-01-21T04:32:47.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.877+0000 7f10f77fe640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-01-21T04:32:47.880 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.877+0000 7f10f77fe640 20 client.4149.objecter dump_active .. 0 homeless 2024-01-21T04:32:47.950 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:47.950+0000 7f4491032040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-21T04:32:47.975 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.975+0000 7f10f77fe640 1 -- 172.21.15.144:0/2627634392 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f110004ec60 con 0x7f11100a8b20 2024-01-21T04:32:47.976 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.975+0000 7f10f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:47.976 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:47.975+0000 7f10f77fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:48.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.046+0000 7f11151a9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e10f1b39-2112-44cf-8045-8b3bc316d07b", "id": 5}] 2024-01-21T04:32:48.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.046+0000 7f11151a9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e10f1b39-2112-44cf-8045-8b3bc316d07b", "id": 5}] 2024-01-21T04:32:48.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.046+0000 7f11151a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:48.047 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.046+0000 7f11151a9640 1 -- 172.21.15.144:0/2627634392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e10f1b39-2112-44cf-8045-8b3bc316d07b", "id": 5} v 0) v1 -- 0x7f1110072860 con 0x7f11100a8b20 2024-01-21T04:32:48.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.075+0000 7f10f77fe640 1 -- 172.21.15.144:0/2627634392 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e10f1b39-2112-44cf-8045-8b3bc316d07b", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f110004ee00 con 0x7f11100a8b20 2024-01-21T04:32:48.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.076+0000 7f10f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e10f1b39-2112-44cf-8045-8b3bc316d07b", "id": 5}] 2024-01-21T04:32:48.076 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.076+0000 7f10f77fe640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:32:48.077 INFO:teuthology.orchestra.run.smithi144.stdout:5 2024-01-21T04:32:48.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.076+0000 7f11151a9640 10 librados: watch_flush enter 2024-01-21T04:32:48.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.076+0000 7f11151a9640 10 librados: watch_flush exit 2024-01-21T04:32:48.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.076+0000 7f11151a9640 20 client.4149.objecter shutdown clearing up homeless session... 2024-01-21T04:32:48.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.076+0000 7f11151a9640 10 client.4149.objecter successfully canceled tick 2024-01-21T04:32:48.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 10 monclient: shutdown 2024-01-21T04:32:48.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:48.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 1 -- 172.21.15.144:0/2627634392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11100a8b20 msgr2=0x7f1110136590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.078 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 1 --2- 172.21.15.144:0/2627634392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11100a8b20 0x7f1110136590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1100034040 tx=0x7f11000046c0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 1 -- 172.21.15.144:0/2627634392 shutdown_connections 2024-01-21T04:32:48.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 1 --2- 172.21.15.144:0/2627634392 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f11100b2310 0x7f1110141570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 1 --2- 172.21.15.144:0/2627634392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11100a9d10 0x7f1110142080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 1 --2- 172.21.15.144:0/2627634392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11100a8b20 0x7f1110136590 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 1 -- 172.21.15.144:0/2627634392 >> 172.21.15.144:0/2627634392 conn(0x7f1110066fb0 msgr2=0x7f1110137510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:48.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 1 -- 172.21.15.144:0/2627634392 shutdown_connections 2024-01-21T04:32:48.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 1 -- 172.21.15.144:0/2627634392 wait complete. 2024-01-21T04:32:48.079 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.077+0000 7f11151a9640 1 librados: shutdown 2024-01-21T04:32:48.088 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new a9b2ce7d-0ae7-4090-b33b-11403d70df1e 6 2024-01-21T04:32:48.187 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.186+0000 7f0f2c032640 10 monclient: get_monmap_and_config 2024-01-21T04:32:48.187 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.186+0000 7f0f2c032640 10 monclient: build_initial_monmap 2024-01-21T04:32:48.187 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.186+0000 7f0f2c032640 10 monclient: monmap: 2024-01-21T04:32:48.187 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:48.187 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:48.187 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:32:48.187705+0000 2024-01-21T04:32:48.187 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:32:48.187705+0000 2024-01-21T04:32:48.187 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:48.188 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.188 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:48.188 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:48.188 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:48.188 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:48.188 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:48.188 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.189 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.187+0000 7f0f2c032640 1 Processor -- start 2024-01-21T04:32:48.189 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.187+0000 7f0f2c032640 1 -- start start 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.187+0000 7f0f2c032640 10 monclient: init 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f240a8a40 addr v1:172.21.15.144:6789/0 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0f240b2210 0x7f0f2413f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 10 monclient(hunting): picked mon.noname-a con 0x7f0f240b2210 addr v2:172.21.15.69:3300/0 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f240a8e20 addr v1:172.21.15.69:6789/0 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f24145700 con 0x7f0f240a8e20 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f240aa9b0 con 0x7f0f240a8a40 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f240aa7b0 con 0x7f0f240b2210 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f2c032640 10 monclient(hunting): authenticate will time out at 1290.504041s 2024-01-21T04:32:48.190 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f29dce640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f0f240a8a40 0x7f0f24145b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:40946/0 (socket says 172.21.15.144:40946) 2024-01-21T04:32:48.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f29dce640 1 -- 172.21.15.144:0/1712466743 learned_addr learned my addr 172.21.15.144:0/1712466743 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-01-21T04:32:48.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.188+0000 7f0f295cd640 1 --2- 172.21.15.144:0/1712466743 >> v2:172.21.15.69:3300/0 conn(0x7f0f240b2210 0x7f0f2413f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:48.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.189+0000 7f0f295cd640 10 monclient(hunting): get_auth_request con 0x7f0f240b2210 auth_method 0 2024-01-21T04:32:48.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.189+0000 7f0f295cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:48.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.189+0000 7f0f295cd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:48.191 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.189+0000 7f0f295cd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:48.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.190+0000 7f0f295cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:48.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.190+0000 7f0f295cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:48.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.190+0000 7f0f295cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:48.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.190+0000 7f0f28dcc640 1 -- 172.21.15.144:0/1712466743 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611729155 0 0) 0x7f0f240aa9b0 con 0x7f0f240a8a40 2024-01-21T04:32:48.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.190+0000 7f0f28dcc640 10 monclient(hunting): _negotiate 2024-01-21T04:32:48.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.190+0000 7f0f28dcc640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:48.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.190+0000 7f0f28dcc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:48.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.190+0000 7f0f28dcc640 10 monclient(hunting): my global_id is 4147 2024-01-21T04:32:48.192 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.190+0000 7f0f28dcc640 1 -- 172.21.15.144:0/1712466743 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f10003520 con 0x7f0f240a8a40 2024-01-21T04:32:48.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.192+0000 7f0f295cd640 10 monclient(hunting): handle_auth_done global_id 4152 payload 274 2024-01-21T04:32:48.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.192+0000 7f0f295cd640 1 -- 172.21.15.144:0/1712466743 >> v1:172.21.15.144:6789/0 conn(0x7f0f240a8a40 legacy=0x7f0f24145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.192+0000 7f0f295cd640 1 -- 172.21.15.144:0/1712466743 >> v1:172.21.15.69:6789/0 conn(0x7f0f240a8e20 legacy=0x7f0f2414c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.192+0000 7f0f295cd640 10 monclient: _finish_hunting 0 2024-01-21T04:32:48.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.192+0000 7f0f295cd640 1 monclient: found mon.noname-a 2024-01-21T04:32:48.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.192+0000 7f0f295cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:48.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.192+0000 7f0f295cd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:48.193 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.192+0000 7f0f295cd640 1 -- 172.21.15.144:0/1712466743 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f240aac90 con 0x7f0f240b2210 2024-01-21T04:32:48.196 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.196+0000 7f0f295cd640 1 --2- 172.21.15.144:0/1712466743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f240b2210 0x7f0f2413f1c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f140097a0 tx=0x7f0f1402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12250348af6160b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:48.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.199+0000 7f0f28dcc640 1 -- 172.21.15.144:0/1712466743 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14030610 con 0x7f0f240b2210 2024-01-21T04:32:48.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.199+0000 7f0f28dcc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:48.200 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.199+0000 7f0f28dcc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.199+0000 7f0f28dcc640 10 monclient: dump: 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.199+0000 7f0f28dcc640 10 monclient: _finish_auth 0 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.199+0000 7f0f28dcc640 10 monclient: _check_auth_tickets 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.199+0000 7f0f28dcc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:48.201 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 5 monclient: authenticate success, global_id 4152 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f28dcc640 1 -- 172.21.15.144:0/1712466743 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f14030770 con 0x7f0f240b2210 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f28dcc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f28dcc640 1 -- 172.21.15.144:0/1712466743 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14030b00 con 0x7f0f240b2210 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 4 set_mon_vals no callback set 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f28dcc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f28dcc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f28dcc640 10 monclient: dump: 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:48.202 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:48.203 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:48.203 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 10 monclient: shutdown 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 1 -- 172.21.15.144:0/1712466743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f240b2210 msgr2=0x7f0f2413f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 1 --2- 172.21.15.144:0/1712466743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f240b2210 0x7f0f2413f1c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0f140097a0 tx=0x7f0f1402b960 comp rx=0 tx=0).stop 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 1 -- 172.21.15.144:0/1712466743 shutdown_connections 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 1 --2- 172.21.15.144:0/1712466743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f240b2210 0x7f0f2413f1c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 1 -- 172.21.15.144:0/1712466743 >> 172.21.15.144:0/1712466743 conn(0x7f0f24066f10 msgr2=0x7f0f240b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 1 -- 172.21.15.144:0/1712466743 shutdown_connections 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.200+0000 7f0f2c032640 1 -- 172.21.15.144:0/1712466743 wait complete. 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.201+0000 7f0f2c032640 10 monclient: build_initial_monmap 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.201+0000 7f0f2c032640 10 monclient: monmap: 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:48.206 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.201+0000 7f0f2c032640 1 librados: starting msgr at 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.201+0000 7f0f2c032640 1 librados: starting objecter 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.201+0000 7f0f2c032640 1 Processor -- start 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.201+0000 7f0f2c032640 1 -- start start 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.201+0000 7f0f2c032640 1 librados: setting wanted keys 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.201+0000 7f0f2c032640 1 librados: calling monclient init 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.201+0000 7f0f2c032640 10 monclient: init 2024-01-21T04:32:48.207 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f240a8a40 0x7f0f24143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 10 monclient(hunting): picked mon.noname-c con 0x7f0f240a8a40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f240a8e20 0x7f0f24141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 10 monclient(hunting): picked mon.noname-a con 0x7f0f240a8e20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f240b2210 0x7f0f24141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 10 monclient(hunting): picked mon.noname-b con 0x7f0f240b2210 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f24144d90 con 0x7f0f240a8e20 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f24140540 con 0x7f0f240b2210 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f24140820 con 0x7f0f240a8a40 2024-01-21T04:32:48.208 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f2c032640 10 monclient(hunting): authenticate will time out at 1290.518363s 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.202+0000 7f0f295cd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f240a8e20 0x7f0f24141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f2a5cf640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f240b2210 0x7f0f24141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f295cd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f240a8e20 0x7f0f24141200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.144:37018/0 (socket says 172.21.15.144:37018) 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f295cd640 1 -- 172.21.15.144:0/1104493050 learned_addr learned my addr 172.21.15.144:0/1104493050 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f295cd640 10 monclient(hunting): get_auth_request con 0x7f0f240a8e20 auth_method 0 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f295cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f295cd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f295cd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f29dce640 1 --2- 172.21.15.144:0/1104493050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f240a8a40 0x7f0f24143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f2a5cf640 10 monclient(hunting): get_auth_request con 0x7f0f240b2210 auth_method 0 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f2a5cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f2a5cf640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f2a5cf640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f295cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:48.209 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f295cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f295cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f29dce640 10 monclient(hunting): get_auth_request con 0x7f0f240a8a40 auth_method 0 2024-01-21T04:32:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f29dce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f29dce640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f29dce640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f2a5cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f2a5cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.203+0000 7f0f2a5cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.204+0000 7f0f29dce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.204+0000 7f0f29dce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:48.210 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.204+0000 7f0f29dce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:48.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.206+0000 7f0f295cd640 10 monclient(hunting): handle_auth_done global_id 4158 payload 824 2024-01-21T04:32:48.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.206+0000 7f0f295cd640 1 -- 172.21.15.144:0/1104493050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f240a8a40 msgr2=0x7f0f24143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.206+0000 7f0f295cd640 1 --2- 172.21.15.144:0/1104493050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f240a8a40 0x7f0f24143fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.206+0000 7f0f295cd640 1 -- 172.21.15.144:0/1104493050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f240b2210 msgr2=0x7f0f24141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.206+0000 7f0f295cd640 1 --2- 172.21.15.144:0/1104493050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f240b2210 0x7f0f24141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.211 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.206+0000 7f0f295cd640 10 monclient: _finish_hunting 0 2024-01-21T04:32:48.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.206+0000 7f0f295cd640 1 monclient: found mon.noname-a 2024-01-21T04:32:48.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.206+0000 7f0f295cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:48.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.206+0000 7f0f295cd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:32:48.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.206+0000 7f0f295cd640 1 -- 172.21.15.144:0/1104493050 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f14009400 con 0x7f0f240a8e20 2024-01-21T04:32:48.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.208+0000 7f0f295cd640 1 --2- 172.21.15.144:0/1104493050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f240a8e20 0x7f0f24141200 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0f140098c0 tx=0x7f0f140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:48.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.208+0000 7f0f1a7fc640 10 client.?.objecter ms_handle_connect 0x7f0f240a8e20 2024-01-21T04:32:48.212 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.208+0000 7f0f1a7fc640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 1 -- 172.21.15.144:0/1104493050 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1403e070 con 0x7f0f240a8e20 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 10 monclient: dump: 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:48.213 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 10 monclient: _finish_auth 0 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 10 monclient: _check_auth_tickets 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 1 -- 172.21.15.144:0/1104493050 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f14039990 con 0x7f0f240a8e20 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f2c032640 5 monclient: authenticate success, global_id 4158 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 1 -- 172.21.15.144:0/1104493050 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14039d40 con 0x7f0f240a8e20 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1a7fc640 10 monclient: dump: 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.214 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f2c032640 10 monclient: _renew_subs 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f1bfff640 4 set_mon_vals no callback set 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f2c032640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.211+0000 7f0f2c032640 1 -- 172.21.15.144:0/1104493050 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f24140b00 con 0x7f0f240a8e20 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.212+0000 7f0f2c032640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.212+0000 7f0f2c032640 10 monclient: _renew_subs 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.212+0000 7f0f2c032640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.212+0000 7f0f2c032640 1 -- 172.21.15.144:0/1104493050 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f24145220 con 0x7f0f240a8e20 2024-01-21T04:32:48.215 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.212+0000 7f0f2c032640 1 librados: init done 2024-01-21T04:32:48.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.212+0000 7f0f2c032640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:48.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.212+0000 7f0f2c032640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:48.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.212+0000 7f0f2c032640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:48.216 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.212+0000 7f0f2c032640 1 -- 172.21.15.144:0/1104493050 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2404e970 con 0x7f0f240a8e20 2024-01-21T04:32:48.311 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.310+0000 7f0f1a7fc640 1 -- 172.21.15.144:0/1104493050 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f0f140557b0 con 0x7f0f240a8e20 2024-01-21T04:32:48.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.314+0000 7f0f1a7fc640 1 -- 172.21.15.144:0/1104493050 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f0f1404be40 con 0x7f0f240a8e20 2024-01-21T04:32:48.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.314+0000 7f0f1a7fc640 10 client.4158.objecter ms_dispatch 0x7f0f24000b90 osd_map(8..8 src has 1..8) v4 2024-01-21T04:32:48.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.314+0000 7f0f1a7fc640 3 client.4158.objecter handle_osd_map got epochs [8,8] > 0 2024-01-21T04:32:48.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.314+0000 7f0f1a7fc640 3 client.4158.objecter handle_osd_map decoding full epoch 8 2024-01-21T04:32:48.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.314+0000 7f0f1a7fc640 20 client.4158.objecter dump_active .. 0 homeless 2024-01-21T04:32:48.315 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.314+0000 7f0f1a7fc640 1 -- 172.21.15.144:0/1104493050 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f0f14034070 con 0x7f0f240a8e20 2024-01-21T04:32:48.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.314+0000 7f0f1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:48.316 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.314+0000 7f0f1a7fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:48.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.387+0000 7f0f2c032640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a9b2ce7d-0ae7-4090-b33b-11403d70df1e", "id": 6}] 2024-01-21T04:32:48.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.387+0000 7f0f2c032640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a9b2ce7d-0ae7-4090-b33b-11403d70df1e", "id": 6}] 2024-01-21T04:32:48.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.387+0000 7f0f2c032640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:32:48.388 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.387+0000 7f0f2c032640 1 -- 172.21.15.144:0/1104493050 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a9b2ce7d-0ae7-4090-b33b-11403d70df1e", "id": 6} v 0) v1 -- 0x7f0f240727d0 con 0x7f0f240a8e20 2024-01-21T04:32:48.435 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.434+0000 7f0f1a7fc640 1 -- 172.21.15.144:0/1104493050 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a9b2ce7d-0ae7-4090-b33b-11403d70df1e", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0f14055bf0 con 0x7f0f240a8e20 2024-01-21T04:32:48.435 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.434+0000 7f0f1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a9b2ce7d-0ae7-4090-b33b-11403d70df1e", "id": 6}] 2024-01-21T04:32:48.435 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.434+0000 7f0f1a7fc640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:32:48.435 INFO:teuthology.orchestra.run.smithi144.stdout:6 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 10 librados: watch_flush enter 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 10 librados: watch_flush exit 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 20 client.4158.objecter shutdown clearing up homeless session... 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 10 client.4158.objecter successfully canceled tick 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 10 monclient: shutdown 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 1 -- 172.21.15.144:0/1104493050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f240a8e20 msgr2=0x7f0f24141200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 1 --2- 172.21.15.144:0/1104493050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f240a8e20 0x7f0f24141200 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0f140098c0 tx=0x7f0f140046c0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 1 -- 172.21.15.144:0/1104493050 shutdown_connections 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 1 --2- 172.21.15.144:0/1104493050 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f240b2210 0x7f0f24141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 1 --2- 172.21.15.144:0/1104493050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f240a8e20 0x7f0f24141200 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.437 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 1 --2- 172.21.15.144:0/1104493050 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0f240a8a40 0x7f0f24143fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 1 -- 172.21.15.144:0/1104493050 >> 172.21.15.144:0/1104493050 conn(0x7f0f24066f10 msgr2=0x7f0f24073df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:48.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 1 -- 172.21.15.144:0/1104493050 shutdown_connections 2024-01-21T04:32:48.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 1 -- 172.21.15.144:0/1104493050 wait complete. 2024-01-21T04:32:48.438 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.435+0000 7f0f2c032640 1 librados: shutdown 2024-01-21T04:32:48.446 DEBUG:teuthology.orchestra.run.smithi144:> sudo ceph --cluster ceph osd new ea6bd6a2-8108-43f6-a833-093408b6fc80 7 2024-01-21T04:32:48.474 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:48.473+0000 7fc4fb42b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-21T04:32:48.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.547+0000 7f48cdbe9640 10 monclient: get_monmap_and_config 2024-01-21T04:32:48.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.547+0000 7f48cdbe9640 10 monclient: build_initial_monmap 2024-01-21T04:32:48.550 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.547+0000 7f48cdbe9640 10 monclient: monmap: 2024-01-21T04:32:48.550 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:48.550 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:48.550 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:32:48.548848+0000 2024-01-21T04:32:48.551 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:32:48.548848+0000 2024-01-21T04:32:48.551 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:48.551 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.551 INFO:teuthology.orchestra.run.smithi144.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:48.551 INFO:teuthology.orchestra.run.smithi144.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:48.551 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:48.552 INFO:teuthology.orchestra.run.smithi144.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:48.552 INFO:teuthology.orchestra.run.smithi144.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:48.552 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.548+0000 7f48cdbe9640 1 Processor -- start 2024-01-21T04:32:48.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.548+0000 7f48cdbe9640 1 -- start start 2024-01-21T04:32:48.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.548+0000 7f48cdbe9640 10 monclient: init 2024-01-21T04:32:48.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:48.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-01-21T04:32:48.556 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48c80a8b20 addr v1:172.21.15.144:6789/0 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a9d10 0x7f48c813f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 10 monclient(hunting): picked mon.noname-b con 0x7f48c80a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48c80b2310 0x7f48c813f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 10 monclient(hunting): picked mon.noname-a con 0x7f48c80b2310 addr v2:172.21.15.69:3300/0 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c81456f0 con 0x7f48c80a8b20 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c80a98c0 con 0x7f48c80b2310 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c813fb00 con 0x7f48c80a9d10 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48cdbe9640 10 monclient(hunting): authenticate will time out at 1290.865250s 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48c77fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f48c80a8b20 0x7f48c8145b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.144:40962/0 (socket says 172.21.15.144:40962) 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48c77fe640 1 -- 172.21.15.144:0/1437328199 learned_addr learned my addr 172.21.15.144:0/1437328199 (peer_addr_for_me v1:172.21.15.144:0/0) 2024-01-21T04:32:48.557 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.549+0000 7f48beffd640 1 --2- 172.21.15.144:0/1437328199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a9d10 0x7f48c813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48c7fff640 1 --2- 172.21.15.144:0/1437328199 >> v2:172.21.15.69:3300/0 conn(0x7f48c80b2310 0x7f48c813f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48beffd640 10 monclient(hunting): get_auth_request con 0x7f48c80a9d10 auth_method 0 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48beffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48beffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48c7fff640 10 monclient(hunting): get_auth_request con 0x7f48c80b2310 auth_method 0 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.550+0000 7f48beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.551+0000 7f48c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.551+0000 7f48c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:48.558 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.551+0000 7f48c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.552+0000 7f48beffd640 10 monclient(hunting): handle_auth_done global_id 4139 payload 274 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.552+0000 7f48beffd640 1 -- 172.21.15.144:0/1437328199 >> v2:172.21.15.69:3300/0 conn(0x7f48c80b2310 msgr2=0x7f48c813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.552+0000 7f48beffd640 1 --2- 172.21.15.144:0/1437328199 >> v2:172.21.15.69:3300/0 conn(0x7f48c80b2310 0x7f48c813f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.552+0000 7f48beffd640 1 -- 172.21.15.144:0/1437328199 >> v1:172.21.15.144:6789/0 conn(0x7f48c80a8b20 legacy=0x7f48c8145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.552+0000 7f48beffd640 10 monclient: _finish_hunting 0 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.552+0000 7f48beffd640 1 monclient: found mon.noname-b 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.552+0000 7f48c6ffd640 1 -- 172.21.15.144:0/1437328199 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587201815 0 0) 0x7f48c81456f0 con 0x7f48c80a8b20 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.552+0000 7f48beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.552+0000 7f48beffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.552+0000 7f48beffd640 1 -- 172.21.15.144:0/1437328199 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c8146290 con 0x7f48c80a9d10 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.552+0000 7f48c6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-21T04:32:48.559 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.554+0000 7f48beffd640 1 --2- 172.21.15.144:0/1437328199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a9d10 0x7f48c813f080 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f48b00097a0 tx=0x7f48b002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cf4b577779af72c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:48.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 1 -- 172.21.15.144:0/1437328199 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b0031830 con 0x7f48c80a9d10 2024-01-21T04:32:48.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:48.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:32:48.563 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 10 monclient: dump: 2024-01-21T04:32:48.563 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:48.563 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:48.563 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 10 monclient: _finish_auth 0 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 10 monclient: _check_auth_tickets 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 1 -- 172.21.15.144:0/1437328199 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b0031990 con 0x7f48c80a9d10 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:48.564 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 1 -- 172.21.15.144:0/1437328199 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b0031d20 con 0x7f48c80a9d10 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48cdbe9640 5 monclient: authenticate success, global_id 4139 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48cdbe9640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48cdbe9640 4 set_mon_vals no callback set 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.557+0000 7f48c6ffd640 10 monclient: dump: 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:48.565 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.558+0000 7f48cdbe9640 10 monclient: shutdown 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.558+0000 7f48cdbe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.558+0000 7f48cdbe9640 1 -- 172.21.15.144:0/1437328199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a9d10 msgr2=0x7f48c813f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.558+0000 7f48cdbe9640 1 --2- 172.21.15.144:0/1437328199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a9d10 0x7f48c813f080 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f48b00097a0 tx=0x7f48b002b8d0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.558+0000 7f48cdbe9640 1 -- 172.21.15.144:0/1437328199 shutdown_connections 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.558+0000 7f48cdbe9640 1 --2- 172.21.15.144:0/1437328199 >> v2:172.21.15.69:3300/0 conn(0x7f48c80b2310 0x7f48c813f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.558+0000 7f48cdbe9640 1 --2- 172.21.15.144:0/1437328199 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a9d10 0x7f48c813f080 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.558+0000 7f48cdbe9640 1 -- 172.21.15.144:0/1437328199 >> 172.21.15.144:0/1437328199 conn(0x7f48c8066fb0 msgr2=0x7f48c80b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.558+0000 7f48cdbe9640 1 -- 172.21.15.144:0/1437328199 shutdown_connections 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.558+0000 7f48cdbe9640 1 -- 172.21.15.144:0/1437328199 wait complete. 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.559+0000 7f48cdbe9640 10 monclient: build_initial_monmap 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.559+0000 7f48cdbe9640 10 monclient: monmap: 2024-01-21T04:32:48.566 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 0 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 0.000000 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:created 0.000000 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.559+0000 7f48cdbe9640 1 librados: starting msgr at 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.559+0000 7f48cdbe9640 1 librados: starting objecter 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.559+0000 7f48cdbe9640 1 Processor -- start 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.559+0000 7f48cdbe9640 1 -- start start 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.559+0000 7f48cdbe9640 1 librados: setting wanted keys 2024-01-21T04:32:48.567 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.559+0000 7f48cdbe9640 1 librados: calling monclient init 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.559+0000 7f48cdbe9640 10 monclient: init 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a8b20 0x7f48c8141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 10 monclient(hunting): picked mon.noname-c con 0x7f48c80a8b20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48c80a9d10 0x7f48c81406e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 10 monclient(hunting): picked mon.noname-a con 0x7f48c80a9d10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c80b2310 0x7f48c81dac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 10 monclient(hunting): picked mon.noname-b con 0x7f48c80b2310 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c8141960 con 0x7f48c80a9d10 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c813fdd0 con 0x7f48c80b2310 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:48.568 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81da730 con 0x7f48c80a8b20 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48cdbe9640 10 monclient(hunting): authenticate will time out at 1290.876256s 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48c77fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a8b20 0x7f48c8141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48c7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c80b2310 0x7f48c81dac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.560+0000 7f48beffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48c80a9d10 0x7f48c81406e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c77fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a8b20 0x7f48c8141420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.144:53944/0 (socket says 172.21.15.144:53944) 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c77fe640 1 -- 172.21.15.144:0/2849554740 learned_addr learned my addr 172.21.15.144:0/2849554740 (peer_addr_for_me v2:172.21.15.144:0/0) 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c77fe640 10 monclient(hunting): get_auth_request con 0x7f48c80a8b20 auth_method 0 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c77fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c7fff640 10 monclient(hunting): get_auth_request con 0x7f48c80b2310 auth_method 0 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:48.569 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48beffd640 10 monclient(hunting): get_auth_request con 0x7f48c80a9d10 auth_method 0 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48beffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48beffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.561+0000 7f48c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.562+0000 7f48beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.562+0000 7f48beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:48.570 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.562+0000 7f48beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:48.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.564+0000 7f48c77fe640 10 monclient(hunting): handle_auth_done global_id 4142 payload 824 2024-01-21T04:32:48.572 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.564+0000 7f48c77fe640 1 -- 172.21.15.144:0/2849554740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c80b2310 msgr2=0x7f48c81dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.564+0000 7f48c77fe640 1 --2- 172.21.15.144:0/2849554740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c80b2310 0x7f48c81dac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.564+0000 7f48c77fe640 1 -- 172.21.15.144:0/2849554740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48c80a9d10 msgr2=0x7f48c81406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.564+0000 7f48c77fe640 1 --2- 172.21.15.144:0/2849554740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48c80a9d10 0x7f48c81406e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.564+0000 7f48c77fe640 10 monclient: _finish_hunting 0 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.564+0000 7f48c77fe640 1 monclient: found mon.noname-c 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.564+0000 7f48c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.564+0000 7f48c77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.564+0000 7f48c77fe640 1 -- 172.21.15.144:0/2849554740 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b0009400 con 0x7f48c80a8b20 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.566+0000 7f48c77fe640 1 --2- 172.21.15.144:0/2849554740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a8b20 0x7f48c8141420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f48b800c3f0 tx=0x7f48b800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.566+0000 7f48c4ff9640 10 client.?.objecter ms_handle_connect 0x7f48c80a8b20 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.566+0000 7f48c4ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48c4ff9640 1 -- 172.21.15.144:0/2849554740 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b80074e0 con 0x7f48c80a8b20 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:48.573 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48c4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48c4ff9640 10 monclient: dump: 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48c4ff9640 10 monclient: _finish_auth 0 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48c4ff9640 10 monclient: _check_auth_tickets 2024-01-21T04:32:48.574 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48cdbe9640 5 monclient: authenticate success, global_id 4142 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48cdbe9640 10 monclient: _renew_subs 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48cdbe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48cdbe9640 1 -- 172.21.15.144:0/2849554740 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c8140c20 con 0x7f48c80a8b20 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48cdbe9640 10 client.4142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48cdbe9640 10 monclient: _renew_subs 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48cdbe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48cdbe9640 1 -- 172.21.15.144:0/2849554740 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c8139f00 con 0x7f48c80a8b20 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.568+0000 7f48cdbe9640 1 librados: init done 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.569+0000 7f48cdbe9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.569+0000 7f48cdbe9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.569+0000 7f48cdbe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:48.575 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.569+0000 7f48cdbe9640 1 -- 172.21.15.144:0/2849554740 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c804e970 con 0x7f48c80a8b20 2024-01-21T04:32:48.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 1 -- 172.21.15.144:0/2849554740 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b80010e0 con 0x7f48c80a8b20 2024-01-21T04:32:48.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:48.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 1 -- 172.21.15.144:0/2849554740 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b80049e0 con 0x7f48c80a8b20 2024-01-21T04:32:48.671 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 10 monclient: dump: 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:epoch 1 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:election_strategy: 1 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T04:32:48.672 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c67fc640 4 set_mon_vals no callback set 2024-01-21T04:32:48.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 1 -- 172.21.15.144:0/2849554740 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f48b801d3f0 con 0x7f48c80a8b20 2024-01-21T04:32:48.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 1 -- 172.21.15.144:0/2849554740 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f48b8013d40 con 0x7f48c80a8b20 2024-01-21T04:32:48.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 10 client.4142.objecter ms_dispatch 0x7f48c8000b90 osd_map(9..9 src has 1..9) v4 2024-01-21T04:32:48.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 3 client.4142.objecter handle_osd_map got epochs [9,9] > 0 2024-01-21T04:32:48.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 3 client.4142.objecter handle_osd_map decoding full epoch 9 2024-01-21T04:32:48.673 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.670+0000 7f48c4ff9640 20 client.4142.objecter dump_active .. 0 homeless 2024-01-21T04:32:48.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.677+0000 7f48c4ff9640 1 -- 172.21.15.144:0/2849554740 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f48b8007640 con 0x7f48c80a8b20 2024-01-21T04:32:48.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.677+0000 7f48c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:48.678 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.677+0000 7f48c4ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:48.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.750+0000 7f48cdbe9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ea6bd6a2-8108-43f6-a833-093408b6fc80", "id": 7}] 2024-01-21T04:32:48.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.750+0000 7f48cdbe9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ea6bd6a2-8108-43f6-a833-093408b6fc80", "id": 7}] 2024-01-21T04:32:48.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.750+0000 7f48cdbe9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:48.751 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.750+0000 7f48cdbe9640 1 -- 172.21.15.144:0/2849554740 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ea6bd6a2-8108-43f6-a833-093408b6fc80", "id": 7} v 0) v1 -- 0x7f48c81da950 con 0x7f48c80a8b20 2024-01-21T04:32:48.754 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:48.753+0000 7fc4fb42b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-21T04:32:48.784 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:48.784+0000 7f4491032040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-21T04:32:48.806 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:48.806+0000 7fc4fb42b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-21T04:32:48.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.823+0000 7f48c4ff9640 1 -- 172.21.15.144:0/2849554740 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ea6bd6a2-8108-43f6-a833-093408b6fc80", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f48b80137b0 con 0x7f48c80a8b20 2024-01-21T04:32:48.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.823+0000 7f48c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ea6bd6a2-8108-43f6-a833-093408b6fc80", "id": 7}] 2024-01-21T04:32:48.824 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.823+0000 7f48c4ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:32:48.824 INFO:teuthology.orchestra.run.smithi144.stdout:7 2024-01-21T04:32:48.825 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 10 librados: watch_flush enter 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 10 librados: watch_flush exit 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 20 client.4142.objecter shutdown clearing up homeless session... 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 10 client.4142.objecter successfully canceled tick 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 10 monclient: shutdown 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 1 -- 172.21.15.144:0/2849554740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a8b20 msgr2=0x7f48c8141420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 1 --2- 172.21.15.144:0/2849554740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a8b20 0x7f48c8141420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f48b800c3f0 tx=0x7f48b800c8b0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 1 -- 172.21.15.144:0/2849554740 shutdown_connections 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 1 --2- 172.21.15.144:0/2849554740 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48c80b2310 0x7f48c81dac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 1 --2- 172.21.15.144:0/2849554740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48c80a9d10 0x7f48c81406e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 1 --2- 172.21.15.144:0/2849554740 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f48c80a8b20 0x7f48c8141420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 1 -- 172.21.15.144:0/2849554740 >> 172.21.15.144:0/2849554740 conn(0x7f48c8066fb0 msgr2=0x7f48c80b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:48.826 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 1 -- 172.21.15.144:0/2849554740 shutdown_connections 2024-01-21T04:32:48.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 1 -- 172.21.15.144:0/2849554740 wait complete. 2024-01-21T04:32:48.827 INFO:teuthology.orchestra.run.smithi144.stderr:2024-01-21T04:32:48.824+0000 7f48cdbe9640 1 librados: shutdown 2024-01-21T04:32:48.835 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T04:32:48.836 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-21T04:32:48.836 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-01-21T04:32:48.839 INFO:tasks.ceph.osd.0:Started 2024-01-21T04:32:48.839 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T04:32:48.839 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-21T04:32:48.839 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-01-21T04:32:48.842 INFO:tasks.ceph.osd.1:Started 2024-01-21T04:32:48.842 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T04:32:48.842 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-21T04:32:48.842 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-01-21T04:32:48.845 INFO:tasks.ceph.osd.2:Started 2024-01-21T04:32:48.846 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T04:32:48.846 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-21T04:32:48.846 DEBUG:teuthology.orchestra.run.smithi069:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-01-21T04:32:48.849 INFO:tasks.ceph.osd.3:Started 2024-01-21T04:32:48.849 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T04:32:48.849 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-21T04:32:48.849 DEBUG:teuthology.orchestra.run.smithi144:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-01-21T04:32:48.880 INFO:tasks.ceph.osd.4:Started 2024-01-21T04:32:48.880 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T04:32:48.880 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-21T04:32:48.880 DEBUG:teuthology.orchestra.run.smithi144:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-01-21T04:32:48.884 INFO:tasks.ceph.osd.5:Started 2024-01-21T04:32:48.884 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T04:32:48.884 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-21T04:32:48.884 DEBUG:teuthology.orchestra.run.smithi144:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-01-21T04:32:48.886 INFO:tasks.ceph.osd.6:Started 2024-01-21T04:32:48.886 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-01-21T04:32:48.887 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-21T04:32:48.887 DEBUG:teuthology.orchestra.run.smithi144:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-01-21T04:32:48.892 INFO:tasks.ceph.osd.7:Started 2024-01-21T04:32:48.892 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-21T04:32:48.922 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:48.921+0000 7fc4fb42b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-21T04:32:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.056+0000 7f85457c0640 10 monclient: get_monmap_and_config 2024-01-21T04:32:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.056+0000 7f85457c0640 10 monclient: build_initial_monmap 2024-01-21T04:32:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.056+0000 7f85457c0640 10 monclient: monmap: 2024-01-21T04:32:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:32:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:32:49.057946+0000 2024-01-21T04:32:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:32:49.057946+0000 2024-01-21T04:32:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:49.059 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:49.059 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:49.059 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:49.059 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:49.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.058+0000 7f85457c0640 1 Processor -- start 2024-01-21T04:32:49.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.058+0000 7f85457c0640 1 -- start start 2024-01-21T04:32:49.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.058+0000 7f85457c0640 10 monclient: init 2024-01-21T04:32:49.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.058+0000 7f85457c0640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:49.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.058+0000 7f85457c0640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-01-21T04:32:49.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.058+0000 7f85457c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f85400a8a40 0x7f8540145b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:49.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.059+0000 7f85457c0640 10 monclient(hunting): picked mon.noname-c con 0x7f85400a8a40 addr v2:172.21.15.144:3300/0 2024-01-21T04:32:49.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.059+0000 7f85457c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85401460a0 addr v1:172.21.15.69:6789/0 2024-01-21T04:32:49.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.059+0000 7f85457c0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854013f8b0 0x7f854014c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:49.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.059+0000 7f85457c0640 10 monclient(hunting): picked mon.noname-b con 0x7f854013f8b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:49.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.059+0000 7f85457c0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85400aa9b0 con 0x7f85401460a0 2024-01-21T04:32:49.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.059+0000 7f85457c0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:49.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.059+0000 7f85457c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85400aa7d0 con 0x7f85400a8a40 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.059+0000 7f85457c0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.059+0000 7f85457c0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85400aac90 con 0x7f854013f8b0 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.059+0000 7f85457c0640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.059+0000 7f85457c0640 10 monclient(hunting): authenticate will time out at 1327.400267s 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854013f8b0 0x7f854014c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853effd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f85400a8a40 0x7f8540145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854013f8b0 0x7f854014c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:58204/0 (socket says 172.21.15.69:58204) 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853f7fe640 1 -- 172.21.15.69:0/1123166223 learned_addr learned my addr 172.21.15.69:0/1123166223 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853f7fe640 10 monclient(hunting): get_auth_request con 0x7f854013f8b0 auth_method 0 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:49.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853effd640 10 monclient(hunting): get_auth_request con 0x7f85400a8a40 auth_method 0 2024-01-21T04:32:49.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:49.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853effd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:49.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.060+0000 7f853effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:49.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.061+0000 7f853f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:49.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.061+0000 7f853f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:49.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.061+0000 7f853f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:49.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.061+0000 7f853effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:49.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.061+0000 7f853effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:49.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.061+0000 7f853effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:49.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.063+0000 7f853f7fe640 10 monclient(hunting): handle_auth_done global_id 4145 payload 274 2024-01-21T04:32:49.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.063+0000 7f853f7fe640 1 -- 172.21.15.69:0/1123166223 >> v2:172.21.15.144:3300/0 conn(0x7f85400a8a40 msgr2=0x7f8540145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:49.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.063+0000 7f853f7fe640 1 --2- 172.21.15.69:0/1123166223 >> v2:172.21.15.144:3300/0 conn(0x7f85400a8a40 0x7f8540145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:49.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.063+0000 7f853f7fe640 1 -- 172.21.15.69:0/1123166223 >> v1:172.21.15.69:6789/0 conn(0x7f85401460a0 legacy=0x7f854013f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:49.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.063+0000 7f853f7fe640 10 monclient: _finish_hunting 0 2024-01-21T04:32:49.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.063+0000 7f853f7fe640 1 monclient: found mon.noname-b 2024-01-21T04:32:49.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.063+0000 7f853f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:49.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.063+0000 7f853f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:32:49.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.063+0000 7f853f7fe640 1 -- 172.21.15.69:0/1123166223 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85400aaf70 con 0x7f854013f8b0 2024-01-21T04:32:49.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.067+0000 7f853f7fe640 1 --2- 172.21.15.69:0/1123166223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854013f8b0 0x7f854014c7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8538009e50 tx=0x7f853802bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1decf134e5ec38ab server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:49.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.075+0000 7f853e7fc640 1 -- 172.21.15.69:0/1123166223 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853802cb20 con 0x7f854013f8b0 2024-01-21T04:32:49.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.075+0000 7f853e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:49.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.075+0000 7f853e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.075+0000 7f853e7fc640 10 monclient: dump: 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f853e7fc640 10 monclient: _finish_auth 0 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f853e7fc640 10 monclient: _check_auth_tickets 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f853e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:49.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f853e7fc640 1 -- 172.21.15.69:0/1123166223 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853802cc80 con 0x7f854013f8b0 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f853e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f853e7fc640 1 -- 172.21.15.69:0/1123166223 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538034610 con 0x7f854013f8b0 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f853e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f853e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f853e7fc640 10 monclient: dump: 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:49.079 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:49.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f85457c0640 5 monclient: authenticate success, global_id 4145 2024-01-21T04:32:49.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f85457c0640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:49.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f85457c0640 4 set_mon_vals no callback set 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f85457c0640 10 monclient: shutdown 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f85457c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f85457c0640 1 -- 172.21.15.69:0/1123166223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854013f8b0 msgr2=0x7f854014c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.076+0000 7f85457c0640 1 --2- 172.21.15.69:0/1123166223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854013f8b0 0x7f854014c7e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8538009e50 tx=0x7f853802bf80 comp rx=0 tx=0).stop 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.077+0000 7f85457c0640 1 -- 172.21.15.69:0/1123166223 shutdown_connections 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.077+0000 7f85457c0640 1 --2- 172.21.15.69:0/1123166223 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f854013f8b0 0x7f854014c7e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.077+0000 7f85457c0640 1 --2- 172.21.15.69:0/1123166223 >> v2:172.21.15.144:3300/0 conn(0x7f85400a8a40 0x7f8540145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.077+0000 7f85457c0640 1 -- 172.21.15.69:0/1123166223 >> 172.21.15.69:0/1123166223 conn(0x7f8540066f50 msgr2=0x7f85400b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.077+0000 7f85457c0640 1 -- 172.21.15.69:0/1123166223 shutdown_connections 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.077+0000 7f85457c0640 1 -- 172.21.15.69:0/1123166223 wait complete. 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.078+0000 7f85457c0640 10 monclient: build_initial_monmap 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.078+0000 7f85457c0640 10 monclient: monmap: 2024-01-21T04:32:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.079+0000 7f85457c0640 1 librados: starting msgr at 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.079+0000 7f85457c0640 1 librados: starting objecter 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.079+0000 7f85457c0640 1 Processor -- start 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.079+0000 7f85457c0640 1 -- start start 2024-01-21T04:32:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.079+0000 7f85457c0640 1 librados: setting wanted keys 2024-01-21T04:32:49.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.079+0000 7f85457c0640 1 librados: calling monclient init 2024-01-21T04:32:49.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.079+0000 7f85457c0640 10 monclient: init 2024-01-21T04:32:49.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:49.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T04:32:49.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85400a8a40 0x7f8540142260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:49.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 10 monclient(hunting): picked mon.noname-b con 0x7f85400a8a40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:49.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854013f8b0 0x7f85401d7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:49.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 10 monclient(hunting): picked mon.noname-a con 0x7f854013f8b0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:49.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85401460a0 0x7f85401d7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:49.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 10 monclient(hunting): picked mon.noname-c con 0x7f85401460a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:49.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:49.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85400abe90 con 0x7f854013f8b0 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85400ac100 con 0x7f85400a8a40 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85400b7cf0 con 0x7f85401460a0 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.080+0000 7f85457c0640 10 monclient(hunting): authenticate will time out at 1327.421826s 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.081+0000 7f853effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85400a8a40 0x7f8540142260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.081+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85401460a0 0x7f85401d7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.081+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85401460a0 0x7f85401d7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:58216/0 (socket says 172.21.15.69:58216) 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.081+0000 7f853f7fe640 1 -- 172.21.15.69:0/2737744455 learned_addr learned my addr 172.21.15.69:0/2737744455 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.081+0000 7f853f7fe640 10 monclient(hunting): get_auth_request con 0x7f85401460a0 auth_method 0 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.081+0000 7f853f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.081+0000 7f853f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.081+0000 7f853f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.082+0000 7f853effd640 10 monclient(hunting): get_auth_request con 0x7f85400a8a40 auth_method 0 2024-01-21T04:32:49.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.082+0000 7f853effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:49.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.082+0000 7f853effd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:49.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.082+0000 7f853effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:49.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.082+0000 7f853f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:49.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.082+0000 7f853f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:49.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.082+0000 7f853f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:49.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.082+0000 7f853effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:49.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.082+0000 7f853effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:49.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.082+0000 7f853effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:49.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.086+0000 7f853f7fe640 10 monclient(hunting): handle_auth_done global_id 4148 payload 824 2024-01-21T04:32:49.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.086+0000 7f853f7fe640 1 -- 172.21.15.69:0/2737744455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85400a8a40 msgr2=0x7f8540142260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:49.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.086+0000 7f853f7fe640 1 --2- 172.21.15.69:0/2737744455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85400a8a40 0x7f8540142260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:49.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.086+0000 7f853f7fe640 1 -- 172.21.15.69:0/2737744455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854013f8b0 msgr2=0x7f85401d7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:49.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.086+0000 7f853f7fe640 1 --2- 172.21.15.69:0/2737744455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854013f8b0 0x7f85401d7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:49.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.086+0000 7f853f7fe640 10 monclient: _finish_hunting 0 2024-01-21T04:32:49.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.086+0000 7f853f7fe640 1 monclient: found mon.noname-c 2024-01-21T04:32:49.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.086+0000 7f853f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:49.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.086+0000 7f853f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:32:49.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.086+0000 7f853f7fe640 1 -- 172.21.15.69:0/2737744455 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8538009b30 con 0x7f85401460a0 2024-01-21T04:32:49.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.090+0000 7f85377fe640 10 client.?.objecter ms_handle_connect 0x7f85401460a0 2024-01-21T04:32:49.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.090+0000 7f85377fe640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:49.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.090+0000 7f853f7fe640 1 --2- 172.21.15.69:0/2737744455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85401460a0 0x7f85401d7540 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8538009e50 tx=0x7f8538004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:49.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.095+0000 7f85377fe640 1 -- 172.21.15.69:0/2737744455 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538009510 con 0x7f85401460a0 2024-01-21T04:32:49.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.095+0000 7f85377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:49.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.095+0000 7f85377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85377fe640 10 monclient: dump: 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85377fe640 10 monclient: _finish_auth 0 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85377fe640 10 monclient: _check_auth_tickets 2024-01-21T04:32:49.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85457c0640 5 monclient: authenticate success, global_id 4148 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85457c0640 10 monclient: _renew_subs 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85457c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85457c0640 1 -- 172.21.15.69:0/2737744455 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85400b8210 con 0x7f85401460a0 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85377fe640 1 -- 172.21.15.69:0/2737744455 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853802c920 con 0x7f85401460a0 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85377fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f853dffb640 4 set_mon_vals no callback set 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85457c0640 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85457c0640 10 monclient: _renew_subs 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85457c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85457c0640 1 -- 172.21.15.69:0/2737744455 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85400b7310 con 0x7f85401460a0 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.096+0000 7f85457c0640 1 librados: init done 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.097+0000 7f85377fe640 1 -- 172.21.15.69:0/2737744455 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538009510 con 0x7f85401460a0 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.097+0000 7f85377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:49.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.097+0000 7f85377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:32:49.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.097+0000 7f85377fe640 10 monclient: dump: 2024-01-21T04:32:49.100 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:32:49.100 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:49.100 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:49.100 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:49.100 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:49.100 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:49.100 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:49.100 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:49.100 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:49.100 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:32:49.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.099+0000 7f85457c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:49.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.100+0000 7f85457c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:49.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.100+0000 7f85457c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:49.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.100+0000 7f85457c0640 1 -- 172.21.15.69:0/2737744455 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85400b7af0 con 0x7f85401460a0 2024-01-21T04:32:49.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.101+0000 7f85377fe640 1 -- 172.21.15.69:0/2737744455 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f853803e450 con 0x7f85401460a0 2024-01-21T04:32:49.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.101+0000 7f85377fe640 1 -- 172.21.15.69:0/2737744455 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f8538051c40 con 0x7f85401460a0 2024-01-21T04:32:49.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.101+0000 7f85377fe640 10 client.4148.objecter ms_dispatch 0x7f8540000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T04:32:49.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.101+0000 7f85377fe640 3 client.4148.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T04:32:49.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.101+0000 7f85377fe640 3 client.4148.objecter handle_osd_map decoding full epoch 10 2024-01-21T04:32:49.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.102+0000 7f85377fe640 20 client.4148.objecter dump_active .. 0 homeless 2024-01-21T04:32:49.207 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:49.207+0000 7f4491032040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-21T04:32:49.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.278+0000 7f85377fe640 1 -- 172.21.15.69:0/2737744455 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7f8538043020 con 0x7f85401460a0 2024-01-21T04:32:49.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.279+0000 7f85377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:49.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.279+0000 7f85377fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:49.298 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:49.297+0000 7f4491032040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-21T04:32:49.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.401+0000 7f85457c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:32:49.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.401+0000 7f85457c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:32:49.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.401+0000 7f85457c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:32:49.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.401+0000 7f85457c0640 1 -- 172.21.15.69:0/2737744455 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f854004e970 con 0x7f85401460a0 2024-01-21T04:32:49.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.419+0000 7f85377fe640 1 -- 172.21.15.69:0/2737744455 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mgrmap(e 2) v1 ==== 44620+0+0 (secure 0 0 0) 0x7f8538051520 con 0x7f85401460a0 2024-01-21T04:32:49.486 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-01-21T04:32:49.485+0000 7f4491032040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-21T04:32:49.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.830+0000 7f85377fe640 1 -- 172.21.15.69:0/2737744455 <== mon.2 v2:172.21.15.69:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f85380333a0 con 0x7f85401460a0 2024-01-21T04:32:49.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.830+0000 7f85377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:32:49.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.830+0000 7f85377fe640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:32:49.832 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:32:49.832 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":10,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T04:32:48.763226+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T04:32:49.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 10 librados: watch_flush enter 2024-01-21T04:32:49.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 10 librados: watch_flush exit 2024-01-21T04:32:49.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 20 client.4148.objecter shutdown clearing up homeless session... 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 10 client.4148.objecter successfully canceled tick 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 10 monclient: shutdown 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 1 -- 172.21.15.69:0/2737744455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85401460a0 msgr2=0x7f85401d7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 1 --2- 172.21.15.69:0/2737744455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85401460a0 0x7f85401d7540 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8538009e50 tx=0x7f8538004a40 comp rx=0 tx=0).stop 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 1 -- 172.21.15.69:0/2737744455 shutdown_connections 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 1 --2- 172.21.15.69:0/2737744455 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f85401460a0 0x7f85401d7540 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 1 --2- 172.21.15.69:0/2737744455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f854013f8b0 0x7f85401d7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 1 --2- 172.21.15.69:0/2737744455 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85400a8a40 0x7f8540142260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 1 -- 172.21.15.69:0/2737744455 >> 172.21.15.69:0/2737744455 conn(0x7f8540066f50 msgr2=0x7f8540139a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 1 -- 172.21.15.69:0/2737744455 shutdown_connections 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.832+0000 7f85357fa640 1 -- 172.21.15.69:0/2737744455 wait complete. 2024-01-21T04:32:49.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:32:49.833+0000 7f85357fa640 1 librados: shutdown 2024-01-21T04:32:49.850 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-21T04:32:49.850 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-21T04:32:50.656 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T04:32:50.798 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.797+0000 7fe84e53a640 10 monclient: get_monmap_and_config 2024-01-21T04:32:50.798 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.797+0000 7fe84e53a640 10 monclient: build_initial_monmap 2024-01-21T04:32:50.798 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.797+0000 7fe84e53a640 10 monclient: monmap: 2024-01-21T04:32:50.798 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:32:50.798 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:50.798 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:32:50.798121+0000 2024-01-21T04:32:50.798 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:32:50.798121+0000 2024-01-21T04:32:50.798 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:50.799 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:50.799 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:50.799 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:50.799 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:50.799 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:50.799 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:50.799 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:50.801 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.798+0000 7fe84e53a640 1 Processor -- start 2024-01-21T04:32:50.801 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.798+0000 7fe84e53a640 1 -- start start 2024-01-21T04:32:50.801 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.798+0000 7fe84e53a640 10 monclient: init 2024-01-21T04:32:50.801 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:50.801 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-01-21T04:32:50.801 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8480a8a60 addr v1:172.21.15.144:6789/0 2024-01-21T04:32:50.801 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe84813fa10 0x7fe8481421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 10 monclient(hunting): picked mon.noname-a con 0x7fe84813fa10 addr v2:172.21.15.69:3300/0 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe848142710 0x7fe848142b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 10 monclient(hunting): picked mon.noname-c con 0x7fe848142710 addr v2:172.21.15.144:3300/0 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8480b1de0 con 0x7fe8480a8a60 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8480a8e40 con 0x7fe84813fa10 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8480aa480 con 0x7fe848142710 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.799+0000 7fe84e53a640 10 monclient(hunting): authenticate will time out at 1329.140451s 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.800+0000 7fe847fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fe8480a8a60 0x7fe84813f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:59032/0 (socket says 172.21.15.69:59032) 2024-01-21T04:32:50.802 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.800+0000 7fe847fff640 1 -- 172.21.15.69:0/3882350671 learned_addr learned my addr 172.21.15.69:0/3882350671 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:32:50.803 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.801+0000 7fe84cad7640 1 --2- 172.21.15.69:0/3882350671 >> v2:172.21.15.144:3300/0 conn(0x7fe848142710 0x7fe848142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:50.803 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.802+0000 7fe84cad7640 10 monclient(hunting): get_auth_request con 0x7fe848142710 auth_method 0 2024-01-21T04:32:50.803 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.802+0000 7fe84cad7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:50.803 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.802+0000 7fe84cad7640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:50.803 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.802+0000 7fe84cad7640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:50.805 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.804+0000 7fe84cad7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:50.805 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.804+0000 7fe84cad7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:50.805 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.804+0000 7fe84cad7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:50.809 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.807+0000 7fe84cad7640 10 monclient(hunting): handle_auth_done global_id 4174 payload 274 2024-01-21T04:32:50.809 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.807+0000 7fe84cad7640 1 -- 172.21.15.69:0/3882350671 >> v2:172.21.15.69:3300/0 conn(0x7fe84813fa10 msgr2=0x7fe8481421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:50.809 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.807+0000 7fe84cad7640 1 --2- 172.21.15.69:0/3882350671 >> v2:172.21.15.69:3300/0 conn(0x7fe84813fa10 0x7fe8481421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:50.809 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.807+0000 7fe84cad7640 1 -- 172.21.15.69:0/3882350671 >> v1:172.21.15.144:6789/0 conn(0x7fe8480a8a60 legacy=0x7fe84813f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:50.809 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.807+0000 7fe84cad7640 10 monclient: _finish_hunting 0 2024-01-21T04:32:50.809 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.807+0000 7fe84cad7640 1 monclient: found mon.noname-c 2024-01-21T04:32:50.809 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.807+0000 7fe84cad7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:50.810 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.807+0000 7fe84cad7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:32:50.810 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.807+0000 7fe84cad7640 1 -- 172.21.15.69:0/3882350671 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8480aa6f0 con 0x7fe848142710 2024-01-21T04:32:50.813 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.813+0000 7fe84cad7640 1 --2- 172.21.15.69:0/3882350671 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe848142710 0x7fe848142b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe8480b1bb0 tx=0x7fe84002be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58533be31fb61b76 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:50.822 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 1 -- 172.21.15.69:0/3882350671 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe84002cc00 con 0x7fe848142710 2024-01-21T04:32:50.822 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:50.822 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:32:50.822 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 10 monclient: dump: 2024-01-21T04:32:50.822 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:32:50.822 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:50.822 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:50.822 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:50.822 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 10 monclient: _finish_auth 0 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 10 monclient: _check_auth_tickets 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 1 -- 172.21.15.69:0/3882350671 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84002cd60 con 0x7fe848142710 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 1 -- 172.21.15.69:0/3882350671 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840034660 con 0x7fe848142710 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:32:50.823 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe846ffd640 10 monclient: dump: 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe84e53a640 5 monclient: authenticate success, global_id 4174 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe84e53a640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:50.824 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.818+0000 7fe84e53a640 4 set_mon_vals no callback set 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.819+0000 7fe84e53a640 10 monclient: shutdown 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.819+0000 7fe84e53a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.819+0000 7fe84e53a640 1 -- 172.21.15.69:0/3882350671 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe848142710 msgr2=0x7fe848142b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.819+0000 7fe84e53a640 1 --2- 172.21.15.69:0/3882350671 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe848142710 0x7fe848142b20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe8480b1bb0 tx=0x7fe84002be50 comp rx=0 tx=0).stop 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.819+0000 7fe84e53a640 1 -- 172.21.15.69:0/3882350671 shutdown_connections 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.819+0000 7fe84e53a640 1 --2- 172.21.15.69:0/3882350671 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe848142710 0x7fe848142b20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.819+0000 7fe84e53a640 1 --2- 172.21.15.69:0/3882350671 >> v2:172.21.15.69:3300/0 conn(0x7fe84813fa10 0x7fe8481421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.819+0000 7fe84e53a640 1 -- 172.21.15.69:0/3882350671 >> 172.21.15.69:0/3882350671 conn(0x7fe8480670b0 msgr2=0x7fe8480b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.819+0000 7fe84e53a640 1 -- 172.21.15.69:0/3882350671 shutdown_connections 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.819+0000 7fe84e53a640 1 -- 172.21.15.69:0/3882350671 wait complete. 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.820+0000 7fe84e53a640 10 monclient: build_initial_monmap 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.820+0000 7fe84e53a640 10 monclient: monmap: 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:50.825 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.821+0000 7fe84e53a640 1 librados: starting msgr at 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.821+0000 7fe84e53a640 1 librados: starting objecter 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.821+0000 7fe84e53a640 1 Processor -- start 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.821+0000 7fe84e53a640 1 -- start start 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.821+0000 7fe84e53a640 1 librados: setting wanted keys 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.821+0000 7fe84e53a640 1 librados: calling monclient init 2024-01-21T04:32:50.826 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.821+0000 7fe84e53a640 10 monclient: init 2024-01-21T04:32:50.830 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:50.830 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T04:32:50.830 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe84813fa10 0x7fe84813e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:50.830 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 10 monclient(hunting): picked mon.noname-c con 0x7fe84813fa10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:50.830 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe848142710 0x7fe8481d7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 10 monclient(hunting): picked mon.noname-b con 0x7fe848142710 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8481d7e60 0x7fe84812fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 10 monclient(hunting): picked mon.noname-a con 0x7fe8481d7e60 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe848141df0 con 0x7fe8481d7e60 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe848141630 con 0x7fe848142710 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8481388c0 con 0x7fe84813fa10 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.822+0000 7fe84e53a640 10 monclient(hunting): authenticate will time out at 1329.163704s 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe8477fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe848142710 0x7fe8481d7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:50.831 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe847fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe84813fa10 0x7fe84813e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe8477fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe848142710 0x7fe8481d7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:34050/0 (socket says 172.21.15.69:34050) 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe8477fe640 1 -- 172.21.15.69:0/790794701 learned_addr learned my addr 172.21.15.69:0/790794701 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe8477fe640 10 monclient(hunting): get_auth_request con 0x7fe848142710 auth_method 0 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe8477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe8477fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe8477fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe847fff640 10 monclient(hunting): get_auth_request con 0x7fe84813fa10 auth_method 0 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe847fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe847fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.823+0000 7fe847fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.824+0000 7fe8477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.824+0000 7fe8477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:50.832 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.824+0000 7fe8477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.824+0000 7fe847fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.824+0000 7fe847fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.824+0000 7fe847fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.828+0000 7fe8477fe640 10 monclient(hunting): handle_auth_done global_id 4177 payload 824 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.828+0000 7fe8477fe640 1 -- 172.21.15.69:0/790794701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe84813fa10 msgr2=0x7fe84813e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.828+0000 7fe8477fe640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe84813fa10 0x7fe84813e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.828+0000 7fe8477fe640 1 -- 172.21.15.69:0/790794701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8481d7e60 msgr2=0x7fe84812fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.828+0000 7fe8477fe640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8481d7e60 0x7fe84812fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.828+0000 7fe8477fe640 10 monclient: _finish_hunting 0 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.828+0000 7fe8477fe640 1 monclient: found mon.noname-b 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.828+0000 7fe8477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.828+0000 7fe8477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:32:50.833 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.828+0000 7fe8477fe640 1 -- 172.21.15.69:0/790794701 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe840009a70 con 0x7fe848142710 2024-01-21T04:32:50.834 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.832+0000 7fe8477fe640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe848142710 0x7fe8481d7920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00b4a0 tx=0x7fe83c00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:50.834 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.832+0000 7fe844ff9640 10 client.?.objecter ms_handle_connect 0x7fe848142710 2024-01-21T04:32:50.834 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.832+0000 7fe844ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:50.841 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe844ff9640 1 -- 172.21.15.69:0/790794701 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c010610 con 0x7fe848142710 2024-01-21T04:32:50.841 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe844ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:50.841 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe844ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:32:50.841 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe844ff9640 10 monclient: dump: 2024-01-21T04:32:50.841 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:32:50.841 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:50.841 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:50.842 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:50.842 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:50.842 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:50.842 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:50.842 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:50.842 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:50.842 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:50.842 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe844ff9640 10 monclient: _finish_auth 0 2024-01-21T04:32:50.842 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe844ff9640 10 monclient: _check_auth_tickets 2024-01-21T04:32:50.842 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe844ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:50.844 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe844ff9640 1 -- 172.21.15.69:0/790794701 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83c010770 con 0x7fe848142710 2024-01-21T04:32:50.844 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe84e53a640 5 monclient: authenticate success, global_id 4177 2024-01-21T04:32:50.844 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe84e53a640 10 monclient: _renew_subs 2024-01-21T04:32:50.844 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe84e53a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:32:50.844 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe84e53a640 1 -- 172.21.15.69:0/790794701 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe848138ba0 con 0x7fe848142710 2024-01-21T04:32:50.844 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe844ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:50.844 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe84e53a640 10 client.4177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.840+0000 7fe8467fc640 4 set_mon_vals no callback set 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.841+0000 7fe84e53a640 10 monclient: _renew_subs 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.841+0000 7fe84e53a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.841+0000 7fe84e53a640 1 -- 172.21.15.69:0/790794701 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe848139160 con 0x7fe848142710 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.841+0000 7fe84e53a640 1 librados: init done 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.841+0000 7fe844ff9640 1 -- 172.21.15.69:0/790794701 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83c010b20 con 0x7fe848142710 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.842+0000 7fe844ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.842+0000 7fe844ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.842+0000 7fe844ff9640 10 monclient: dump: 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:50.845 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:50.846 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:50.846 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:50.846 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:50.846 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:50.846 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.843+0000 7fe84e53a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:50.846 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.843+0000 7fe84e53a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:50.846 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.843+0000 7fe84e53a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:32:50.846 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.843+0000 7fe84e53a640 1 -- 172.21.15.69:0/790794701 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe84806b500 con 0x7fe848142710 2024-01-21T04:32:50.861 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:50.860+0000 7fe844ff9640 1 -- 172.21.15.69:0/790794701 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 2) v1 ==== 44620+0+0 (secure 0 0 0) 0x7fe83c010c80 con 0x7fe848142710 2024-01-21T04:32:51.152 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.151+0000 7fe844ff9640 1 -- 172.21.15.69:0/790794701 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe83c00f360 con 0x7fe848142710 2024-01-21T04:32:51.152 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.151+0000 7fe844ff9640 10 client.4177.objecter ms_dispatch 0x7fe848000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T04:32:51.152 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.151+0000 7fe844ff9640 3 client.4177.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T04:32:51.153 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.151+0000 7fe844ff9640 3 client.4177.objecter handle_osd_map decoding full epoch 10 2024-01-21T04:32:51.153 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.151+0000 7fe844ff9640 20 client.4177.objecter dump_active .. 0 homeless 2024-01-21T04:32:51.161 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.161+0000 7fe844ff9640 1 -- 172.21.15.69:0/790794701 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72921 (secure 0 0 0) 0x7fe83c0183b0 con 0x7fe848142710 2024-01-21T04:32:51.161 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.161+0000 7fe844ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:51.162 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.161+0000 7fe844ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:51.287 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.285+0000 7fe84e53a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:32:51.287 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.286+0000 7fe84e53a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:32:51.287 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.286+0000 7fe84e53a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:32:51.287 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.286+0000 7fe84e53a640 1 -- 172.21.15.69:0/790794701 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe848075c30 con 0x7fe848142710 2024-01-21T04:32:51.602 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.601+0000 7fe844ff9640 1 -- 172.21.15.69:0/790794701 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mgrmap(e 3) v1 ==== 89036+0+0 (secure 0 0 0) 0x7fe83c0239a0 con 0x7fe848142710 2024-01-21T04:32:51.602 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.602+0000 7fe844ff9640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe83406ce40 0x7fe83406f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:51.603 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.602+0000 7fe847fff640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe83406ce40 0x7fe83406f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:51.603 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.602+0000 7fe847fff640 10 monclient: get_auth_request con 0x7fe83406ce40 auth_method 0 2024-01-21T04:32:51.603 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.602+0000 7fe844ff9640 10 client.4177.objecter ms_handle_connect 0x7fe83406ce40 2024-01-21T04:32:51.603 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.602+0000 7fe847fff640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe83406ce40 0x7fe83406f230 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe838005880 tx=0x7fe838009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:51.625 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:51.625+0000 7fc49c0fa640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-21T04:32:51.764 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.763+0000 7fe844ff9640 1 -- 172.21.15.69:0/790794701 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fe83c056590 con 0x7fe848142710 2024-01-21T04:32:51.764 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.763+0000 7fe844ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:32:51.764 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.763+0000 7fe844ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:32:51.765 INFO:teuthology.misc.health.smithi069.stdout: 2024-01-21T04:32:51.765 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T04:32:48.763226+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T04:32:51.766 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.764+0000 7fe84e53a640 10 librados: watch_flush enter 2024-01-21T04:32:51.766 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.764+0000 7fe84e53a640 10 librados: watch_flush exit 2024-01-21T04:32:51.766 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.764+0000 7fe84e53a640 20 client.4177.objecter shutdown clearing up homeless session... 2024-01-21T04:32:51.766 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.764+0000 7fe84e53a640 10 client.4177.objecter successfully canceled tick 2024-01-21T04:32:51.766 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.764+0000 7fe84e53a640 1 -- 172.21.15.69:0/790794701 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe83406ce40 msgr2=0x7fe83406f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.764+0000 7fe84e53a640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe83406ce40 0x7fe83406f230 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe838005880 tx=0x7fe838009040 comp rx=0 tx=0).stop 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 10 monclient: shutdown 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 1 -- 172.21.15.69:0/790794701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe848142710 msgr2=0x7fe8481d7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe848142710 0x7fe8481d7920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00b4a0 tx=0x7fe83c00b960 comp rx=0 tx=0).stop 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 1 -- 172.21.15.69:0/790794701 shutdown_connections 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe83406ce40 0x7fe83406f230 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8481d7e60 0x7fe84812fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe848142710 0x7fe8481d7920 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00b4a0 tx=0x7fe83c00b960 comp rx=0 tx=0).stop 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 1 --2- 172.21.15.69:0/790794701 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe84813fa10 0x7fe84813e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 1 -- 172.21.15.69:0/790794701 >> 172.21.15.69:0/790794701 conn(0x7fe8480a8a60 msgr2=0x7fe84813f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 1 -- 172.21.15.69:0/790794701 shutdown_connections 2024-01-21T04:32:51.767 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 1 -- 172.21.15.69:0/790794701 wait complete. 2024-01-21T04:32:51.768 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:51.765+0000 7fe84e53a640 1 librados: shutdown 2024-01-21T04:32:51.780 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-21T04:32:51.856 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T04:32:51.855+0000 7fc49c0fa640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-21T04:32:56.282 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:07.313 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.301 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:07.346 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.301 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:07.333 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.344 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:07.375 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.365 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:07.410 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.374 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:07.405 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.585 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:07.642 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.599 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:07.684 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.637 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:07.731 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.644 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:07.701 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.650 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:07.725 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.661 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:07.747 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.697 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:07.792 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.742 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:07.816 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.822 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:07.899 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.881 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:07.959 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:56.886 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T04:32:56.879+0000 5dc2300 -1 Falling back to public interface 2024-01-21T04:32:56.927 INFO:tasks.ceph.osd.7.smithi144.stderr:2024-01-21T04:32:56.920+0000 5dc2300 -1 Falling back to public interface 2024-01-21T04:32:56.931 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:32:56.925+0000 5dc2300 -1 Falling back to public interface 2024-01-21T04:32:57.228 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:32:57.222+0000 5dc2300 -1 Falling back to public interface 2024-01-21T04:32:57.235 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:32:57.225+0000 5dc2300 -1 Falling back to public interface 2024-01-21T04:32:57.265 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:32:57.255+0000 5dc2300 -1 Falling back to public interface 2024-01-21T04:32:57.289 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:32:57.281+0000 5dc2300 -1 Falling back to public interface 2024-01-21T04:32:57.336 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:08.367 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:57.370 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:08.415 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:57.376 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:08.407 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:57.453 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:32:57.445+0000 5dc2300 -1 Falling back to public interface 2024-01-21T04:32:57.682 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:08.739 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:57.694 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:08.779 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:57.702 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:08.796 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:57.710 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:08.741 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:57.730 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:08.804 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:57.757 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:08.802 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:57.771 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:08.802 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:57.919 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:08.997 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.037 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:09.068 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.060 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:09.146 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.075 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:09.169 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.075 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:09.132 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.079 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:09.124 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.111 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:09.186 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.144 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:09.175 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.181 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:09.255 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.287 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:09.365 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.305 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:09.336 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.351 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:09.396 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.395 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:09.489 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.414 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:09.500 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.440 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:09.471 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.448 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:09.522 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.453 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:09.510 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.572 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:09.603 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.588 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T04:32:58.597 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:09.675 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.615 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:09.660 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.676 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.675+0000 7f9ce2eec640 10 monclient: get_monmap_and_config 2024-01-21T04:32:58.676 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.675+0000 7f9ce2eec640 10 monclient: build_initial_monmap 2024-01-21T04:32:58.676 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.675+0000 7f9ce2eec640 10 monclient: monmap: 2024-01-21T04:32:58.676 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:32:58.676 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:58.676 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:32:58.676163+0000 2024-01-21T04:32:58.676 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:32:58.676163+0000 2024-01-21T04:32:58.676 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:58.676 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:58.676 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:32:58.677 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:32:58.677 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:32:58.677 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:32:58.677 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:32:58.677 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:58.679 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.675+0000 7f9ce2eec640 1 Processor -- start 2024-01-21T04:32:58.680 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.675+0000 7f9ce2eec640 1 -- start start 2024-01-21T04:32:58.680 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 10 monclient: init 2024-01-21T04:32:58.680 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:58.680 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-01-21T04:32:58.680 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9cdc0a8a60 0x7f9cdc13f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:58.680 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 10 monclient(hunting): picked mon.noname-c con 0x7f9cdc0a8a60 addr v2:172.21.15.144:3300/0 2024-01-21T04:32:58.680 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9cdc13f840 0x7f9cdc1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:58.680 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 10 monclient(hunting): picked mon.noname-a con 0x7f9cdc13f840 addr v2:172.21.15.69:3300/0 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9cdc142710 addr v1:172.21.15.69:6789/0 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cdc0b1de0 con 0x7f9cdc142710 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cdc0a8e40 con 0x7f9cdc13f840 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cdc13fc70 con 0x7f9cdc0a8a60 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.676+0000 7f9ce2eec640 10 monclient(hunting): authenticate will time out at 1337.017836s 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9ce0c88640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f9cdc0a8a60 0x7f9cdc13f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9cdbfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9cdc13f840 0x7f9cdc1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9ce1489640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9cdc142710 0x7f9cdc1455a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37948/0 (socket says 172.21.15.69:37948) 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9ce1489640 1 -- 172.21.15.69:0/4187576309 learned_addr learned my addr 172.21.15.69:0/4187576309 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:32:58.681 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9ce0c88640 10 monclient(hunting): get_auth_request con 0x7f9cdc0a8a60 auth_method 0 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9ce0c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9ce0c88640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9ce0c88640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9cdbfff640 10 monclient(hunting): get_auth_request con 0x7f9cdc13f840 auth_method 0 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9cdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9cdbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.677+0000 7f9cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.678+0000 7f9ce0c88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.678+0000 7f9ce0c88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.678+0000 7f9ce0c88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.678+0000 7f9cdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.678+0000 7f9cdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:58.682 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.678+0000 7f9cdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:58.683 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.679+0000 7f9ce0c88640 10 monclient(hunting): handle_auth_done global_id 4201 payload 274 2024-01-21T04:32:58.683 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.679+0000 7f9ce0c88640 1 -- 172.21.15.69:0/4187576309 >> v2:172.21.15.69:3300/0 conn(0x7f9cdc13f840 msgr2=0x7f9cdc1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:32:58.683 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.679+0000 7f9ce0c88640 1 --2- 172.21.15.69:0/4187576309 >> v2:172.21.15.69:3300/0 conn(0x7f9cdc13f840 0x7f9cdc1421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:58.683 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.679+0000 7f9ce0c88640 1 -- 172.21.15.69:0/4187576309 >> v1:172.21.15.69:6789/0 conn(0x7f9cdc142710 legacy=0x7f9cdc1455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:58.683 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.679+0000 7f9ce0c88640 10 monclient: _finish_hunting 0 2024-01-21T04:32:58.683 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.679+0000 7f9ce0c88640 1 monclient: found mon.noname-c 2024-01-21T04:32:58.683 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.679+0000 7f9ce0c88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:58.684 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.679+0000 7f9ce0c88640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:32:58.684 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.679+0000 7f9ce0c88640 1 -- 172.21.15.69:0/4187576309 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc0aa480 con 0x7f9cdc0a8a60 2024-01-21T04:32:58.684 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:09.774 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.684 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.681+0000 7f9ce0c88640 1 --2- 172.21.15.69:0/4187576309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cdc0a8a60 0x7f9cdc13f300 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0b53b0 tx=0x7f9cd002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bf40dbb17935970 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:58.686 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 1 -- 172.21.15.69:0/4187576309 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd002e070 con 0x7f9cdc0a8a60 2024-01-21T04:32:58.686 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 10 monclient: dump: 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 10 monclient: _finish_auth 0 2024-01-21T04:32:58.687 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 10 monclient: _check_auth_tickets 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 1 -- 172.21.15.69:0/4187576309 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd00315a0 con 0x7f9cdc0a8a60 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 5 monclient: authenticate success, global_id 4201 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 1 -- 172.21.15.69:0/4187576309 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0031930 con 0x7f9cdc0a8a60 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 10 monclient: get_monmap_and_config success 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 4 set_mon_vals no callback set 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9cdb7fe640 10 monclient: dump: 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:58.688 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 10 monclient: shutdown 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 1 -- 172.21.15.69:0/4187576309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cdc0a8a60 msgr2=0x7f9cdc13f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 1 --2- 172.21.15.69:0/4187576309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cdc0a8a60 0x7f9cdc13f300 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0b53b0 tx=0x7f9cd002b840 comp rx=0 tx=0).stop 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 1 -- 172.21.15.69:0/4187576309 shutdown_connections 2024-01-21T04:32:58.689 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 1 --2- 172.21.15.69:0/4187576309 >> v2:172.21.15.69:3300/0 conn(0x7f9cdc13f840 0x7f9cdc1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:58.690 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 1 --2- 172.21.15.69:0/4187576309 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cdc0a8a60 0x7f9cdc13f300 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:58.690 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 1 -- 172.21.15.69:0/4187576309 >> 172.21.15.69:0/4187576309 conn(0x7f9cdc0670b0 msgr2=0x7f9cdc0b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:58.690 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 1 -- 172.21.15.69:0/4187576309 shutdown_connections 2024-01-21T04:32:58.690 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.685+0000 7f9ce2eec640 1 -- 172.21.15.69:0/4187576309 wait complete. 2024-01-21T04:32:58.693 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.686+0000 7f9ce2eec640 10 monclient: build_initial_monmap 2024-01-21T04:32:58.693 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.686+0000 7f9ce2eec640 10 monclient: monmap: 2024-01-21T04:32:58.693 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:32:58.693 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:32:58.694 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-01-21T04:32:58.694 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-01-21T04:32:58.694 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:32:58.694 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:58.694 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:32:58.694 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:32:58.694 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:32:58.694 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:58.694 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.686+0000 7f9ce2eec640 1 librados: starting msgr at 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.686+0000 7f9ce2eec640 1 librados: starting objecter 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.686+0000 7f9ce2eec640 1 Processor -- start 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.686+0000 7f9ce2eec640 1 -- start start 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.686+0000 7f9ce2eec640 1 librados: setting wanted keys 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.686+0000 7f9ce2eec640 1 librados: calling monclient init 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.686+0000 7f9ce2eec640 10 monclient: init 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 10 monclient: _reopen_session rank -1 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cdc13f840 0x7f9cdc13b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 10 monclient(hunting): picked mon.noname-b con 0x7f9cdc13f840 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cdc142710 0x7f9cdc13bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 10 monclient(hunting): picked mon.noname-a con 0x7f9cdc142710 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:32:58.695 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cdc140180 0x7f9cdc1d7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 10 monclient(hunting): picked mon.noname-c con 0x7f9cdc140180 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc1d7400 con 0x7f9cdc142710 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc143e90 con 0x7f9cdc13f840 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 10 monclient(hunting): start opening mon connection 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc13c250 con 0x7f9cdc140180 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 10 monclient(hunting): _renew_subs 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.687+0000 7f9ce2eec640 10 monclient(hunting): authenticate will time out at 1337.028764s 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9cdbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cdc142710 0x7f9cdc13bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9ce1489640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cdc140180 0x7f9cdc1d7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9ce0c88640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cdc13f840 0x7f9cdc13b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:58.696 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9cdbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cdc142710 0x7f9cdc13bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56654/0 (socket says 172.21.15.69:56654) 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9cdbfff640 1 -- 172.21.15.69:0/2307047056 learned_addr learned my addr 172.21.15.69:0/2307047056 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9cdbfff640 10 monclient(hunting): get_auth_request con 0x7f9cdc142710 auth_method 0 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9cdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9cdbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9ce1489640 10 monclient(hunting): get_auth_request con 0x7f9cdc140180 auth_method 0 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9ce1489640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9ce1489640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9ce1489640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9ce0c88640 10 monclient(hunting): get_auth_request con 0x7f9cdc13f840 auth_method 0 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9ce0c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9ce0c88640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.688+0000 7f9ce0c88640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:32:58.697 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.689+0000 7f9cdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:58.698 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.689+0000 7f9cdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:58.698 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.689+0000 7f9cdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:58.698 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.689+0000 7f9ce0c88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:58.698 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.689+0000 7f9ce0c88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:58.698 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.689+0000 7f9ce0c88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:58.698 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.689+0000 7f9ce1489640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:32:58.698 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.689+0000 7f9ce1489640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:32:58.698 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.689+0000 7f9ce1489640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:32:58.699 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.691+0000 7f9ce0c88640 10 monclient(hunting): handle_auth_done global_id 4204 payload 824 2024-01-21T04:32:58.699 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.691+0000 7f9ce0c88640 1 -- 172.21.15.69:0/2307047056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cdc140180 msgr2=0x7f9cdc1d7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:58.699 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.691+0000 7f9ce0c88640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cdc140180 0x7f9cdc1d7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:58.699 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.691+0000 7f9ce0c88640 1 -- 172.21.15.69:0/2307047056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cdc142710 msgr2=0x7f9cdc13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:58.699 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.691+0000 7f9ce0c88640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cdc142710 0x7f9cdc13bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:58.699 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.691+0000 7f9ce0c88640 10 monclient: _finish_hunting 0 2024-01-21T04:32:58.699 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.691+0000 7f9ce0c88640 1 monclient: found mon.noname-b 2024-01-21T04:32:58.699 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.691+0000 7f9ce0c88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:32:58.699 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.691+0000 7f9ce0c88640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:32:58.699 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.691+0000 7f9ce0c88640 1 -- 172.21.15.69:0/2307047056 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd0009400 con 0x7f9cdc13f840 2024-01-21T04:32:58.701 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.693+0000 7f9ce0c88640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cdc13f840 0x7f9cdc13b150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0033000 tx=0x7f9cd00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:58.701 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.693+0000 7f9cd97fa640 10 client.?.objecter ms_handle_connect 0x7f9cdc13f840 2024-01-21T04:32:58.702 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.693+0000 7f9cd97fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:32:58.702 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.696+0000 7f9cd97fa640 1 -- 172.21.15.69:0/2307047056 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0032840 con 0x7f9cdc13f840 2024-01-21T04:32:58.702 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.696+0000 7f9cd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:58.702 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.696+0000 7f9cd97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:32:58.703 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.696+0000 7f9cd97fa640 10 monclient: dump: 2024-01-21T04:32:58.703 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:32:58.703 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:58.703 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:58.704 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:58.704 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:58.704 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:58.704 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:58.705 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:58.705 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:58.705 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:58.705 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9cd97fa640 10 monclient: _finish_auth 0 2024-01-21T04:32:58.705 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9cd97fa640 10 monclient: _check_auth_tickets 2024-01-21T04:32:58.705 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9cd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:32:58.705 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9cd97fa640 1 -- 172.21.15.69:0/2307047056 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd00329a0 con 0x7f9cdc13f840 2024-01-21T04:32:58.705 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 5 monclient: authenticate success, global_id 4204 2024-01-21T04:32:58.705 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 10 monclient: _renew_subs 2024-01-21T04:32:58.705 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:32:58.705 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 1 -- 172.21.15.69:0/2307047056 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cdc1d7dd0 con 0x7f9cdc13f840 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9cd97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9cd97fa640 1 -- 172.21.15.69:0/2307047056 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0032d30 con 0x7f9cdc13f840 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9cd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9cd97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9cd97fa640 10 monclient: dump: 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:32:58.706 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9cdaffd640 4 set_mon_vals no callback set 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 10 monclient: _renew_subs 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 1 -- 172.21.15.69:0/2307047056 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cdc1da740 con 0x7f9cdc13f840 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 1 librados: init done 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:32:58.707 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.697+0000 7f9ce2eec640 1 -- 172.21.15.69:0/2307047056 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cdc13c3e0 con 0x7f9cdc13f840 2024-01-21T04:32:58.708 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:09.791 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.715+0000 7f9cd97fa640 1 -- 172.21.15.69:0/2307047056 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7f9cd004e440 con 0x7f9cdc13f840 2024-01-21T04:32:58.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.715+0000 7f9cd97fa640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f9cc006c820 0x7f9cc006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:32:58.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.716+0000 7f9cdbfff640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f9cc006c820 0x7f9cc006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:32:58.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.716+0000 7f9cdbfff640 10 monclient: get_auth_request con 0x7f9cc006c820 auth_method 0 2024-01-21T04:32:58.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.716+0000 7f9cdbfff640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f9cc006c820 0x7f9cc006ece0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc13ae70 tx=0x7f9cc8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:32:58.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.716+0000 7f9cd97fa640 10 client.4204.objecter ms_handle_connect 0x7f9cc006c820 2024-01-21T04:32:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.717+0000 7f9cd97fa640 1 -- 172.21.15.69:0/2307047056 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9cd007a240 con 0x7f9cdc13f840 2024-01-21T04:32:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.717+0000 7f9cd97fa640 10 client.4204.objecter ms_dispatch 0x7f9cdc000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T04:32:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.717+0000 7f9cd97fa640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T04:32:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.717+0000 7f9cd97fa640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2024-01-21T04:32:58.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.717+0000 7f9cd97fa640 20 client.4204.objecter dump_active .. 0 homeless 2024-01-21T04:32:58.721 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:09.795 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.731 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:09.762 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.748 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:09.802 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.842 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:09.873 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.865 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:09.943 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.884 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:09.929 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.944 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.038 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.955 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.955+0000 7f9cd97fa640 1 -- 172.21.15.69:0/2307047056 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9cd0076580 con 0x7f9cdc13f840 2024-01-21T04:32:58.956 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.955+0000 7f9cd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:32:58.956 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:58.955+0000 7f9cd97fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:32:58.985 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.060 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:58.997 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.083 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.021 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:10.053 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.047 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:10.104 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.114 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:10.145 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.129 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.207 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.132 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.131+0000 7f9ce2eec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:32:59.132 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.131+0000 7f9ce2eec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:32:59.132 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.131+0000 7f9ce2eec640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:32:59.132 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.131+0000 7f9ce2eec640 1 -- 172.21.15.69:0/2307047056 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cdc075c30 con 0x7f9cdc13f840 2024-01-21T04:32:59.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.147+0000 7f9cd97fa640 1 -- 172.21.15.69:0/2307047056 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f9cd0079ba0 con 0x7f9cdc13f840 2024-01-21T04:32:59.148 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.147+0000 7f9cd97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:32:59.148 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.147+0000 7f9cd97fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:32:59.148 INFO:teuthology.misc.health.smithi069.stdout: 2024-01-21T04:32:59.148 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T04:32:48.763226+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T04:32:59.149 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:10.194 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 10 librados: watch_flush enter 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 10 librados: watch_flush exit 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 20 client.4204.objecter shutdown clearing up homeless session... 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 10 client.4204.objecter successfully canceled tick 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 -- 172.21.15.69:0/2307047056 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f9cc006c820 msgr2=0x7f9cc006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f9cc006c820 0x7f9cc006ece0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc13ae70 tx=0x7f9cc8009190 comp rx=0 tx=0).stop 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 10 monclient: shutdown 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 -- 172.21.15.69:0/2307047056 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cdc13f840 msgr2=0x7f9cdc13b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cdc13f840 0x7f9cdc13b150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0033000 tx=0x7f9cd00046c0 comp rx=0 tx=0).stop 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 -- 172.21.15.69:0/2307047056 shutdown_connections 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f9cc006c820 0x7f9cc006ece0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:59.151 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f9cdc140180 0x7f9cdc1d7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:59.152 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cdc142710 0x7f9cdc13bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:59.152 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 --2- 172.21.15.69:0/2307047056 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9cdc13f840 0x7f9cdc13b150 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:32:59.152 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 -- 172.21.15.69:0/2307047056 >> 172.21.15.69:0/2307047056 conn(0x7f9cdc0a8a60 msgr2=0x7f9cdc139180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:32:59.152 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 -- 172.21.15.69:0/2307047056 shutdown_connections 2024-01-21T04:32:59.152 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 -- 172.21.15.69:0/2307047056 wait complete. 2024-01-21T04:32:59.152 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:32:59.149+0000 7f9ce2eec640 1 librados: shutdown 2024-01-21T04:32:59.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-21T04:32:59.224 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.318 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.252 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.326 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.289 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.374 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.310 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:10.341 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.339 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:10.396 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.388 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:10.419 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.393 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.471 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.420 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:10.465 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.489 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.583 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.518 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.593 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.578 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.663 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.598 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:10.628 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.638 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:10.695 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.657 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.735 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.658 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:10.689 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.690 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:10.735 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.753 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:10.847 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.789 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:10.863 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.869 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:10.954 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.884 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:10.915 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.923 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:10.999 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.935 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:10.991 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:32:59.982 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:11.039 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.019 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:11.113 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.136 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:11.167 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.157 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:11.202 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.159 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.245 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.190 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:11.221 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.190 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:11.268 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.232 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:11.277 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.261 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:11.335 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.314 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:11.389 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.420 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:11.451 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.480 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:11.511 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.497 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:11.591 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.541 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:11.598 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.588 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:11.682 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.608 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:11.665 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.690 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:11.768 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.747 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:11.822 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.754 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.840 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:00.827 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:11.912 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:05.655 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:16.686 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:05.924 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:16.998 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:05.967 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T04:33:05.967 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:17.024 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:05.998 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:17.043 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:06.056 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:17.150 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:06.079 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.078+0000 7f6b66621640 10 monclient: get_monmap_and_config 2024-01-21T04:33:06.079 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.078+0000 7f6b66621640 10 monclient: build_initial_monmap 2024-01-21T04:33:06.079 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.078+0000 7f6b66621640 10 monclient: monmap: 2024-01-21T04:33:06.079 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:06.080 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:06.080 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:33:06.079446+0000 2024-01-21T04:33:06.080 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:33:06.079446+0000 2024-01-21T04:33:06.080 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:06.080 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:06.080 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:06.080 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:06.080 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:06.080 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:06.080 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:06.080 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:06.083 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.079+0000 7f6b66621640 1 Processor -- start 2024-01-21T04:33:06.083 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.079+0000 7f6b66621640 1 -- start start 2024-01-21T04:33:06.083 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.079+0000 7f6b66621640 10 monclient: init 2024-01-21T04:33:06.083 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:06.083 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-01-21T04:33:06.083 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b600a8a60 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:06.083 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b6013fa10 0x7f6b601421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:06.083 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 10 monclient(hunting): picked mon.noname-c con 0x7f6b6013fa10 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:06.083 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 0x7f6b60142b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 10 monclient(hunting): picked mon.noname-b con 0x7f6b60142710 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b600b1de0 con 0x7f6b600a8a60 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b600a8e40 con 0x7f6b6013fa10 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b600aa480 con 0x7f6b60142710 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.080+0000 7f6b66621640 10 monclient(hunting): authenticate will time out at 1344.421951s 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.081+0000 7f6b64bbe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 0x7f6b60142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.081+0000 7f6b64bbe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 0x7f6b60142b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37918/0 (socket says 172.21.15.69:37918) 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.081+0000 7f6b64bbe640 1 -- 172.21.15.69:0/2475732137 learned_addr learned my addr 172.21.15.69:0/2475732137 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.081+0000 7f6b64bbe640 10 monclient(hunting): get_auth_request con 0x7f6b60142710 auth_method 0 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.081+0000 7f6b64bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:06.084 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.081+0000 7f6b64bbe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:06.085 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.081+0000 7f6b64bbe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.082+0000 7f6b64bbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.082+0000 7f6b64bbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.082+0000 7f6b64bbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.084+0000 7f6b64bbe640 10 monclient(hunting): handle_auth_done global_id 4190 payload 274 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.084+0000 7f6b64bbe640 1 -- 172.21.15.69:0/2475732137 >> v2:172.21.15.144:3300/0 conn(0x7f6b6013fa10 msgr2=0x7f6b601421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.084+0000 7f6b64bbe640 1 --2- 172.21.15.69:0/2475732137 >> v2:172.21.15.144:3300/0 conn(0x7f6b6013fa10 0x7f6b601421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.084+0000 7f6b64bbe640 1 -- 172.21.15.69:0/2475732137 >> v1:172.21.15.144:6789/0 conn(0x7f6b600a8a60 legacy=0x7f6b6013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.084+0000 7f6b64bbe640 10 monclient: _finish_hunting 0 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.084+0000 7f6b64bbe640 1 monclient: found mon.noname-b 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.084+0000 7f6b64bbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.084+0000 7f6b64bbe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:33:06.086 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.084+0000 7f6b64bbe640 1 -- 172.21.15.69:0/2475732137 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b600aa6f0 con 0x7f6b60142710 2024-01-21T04:33:06.087 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.087+0000 7f6b64bbe640 1 --2- 172.21.15.69:0/2475732137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 0x7f6b60142b40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009e50 tx=0x7f6b5802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a7ed78df0f94c0f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 1 -- 172.21.15.69:0/2475732137 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5802c9e0 con 0x7f6b60142710 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 10 monclient: dump: 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:06.135 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 10 monclient: _finish_auth 0 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 1 -- 172.21.15.69:0/2475732137 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5802cb40 con 0x7f6b60142710 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 1 -- 172.21.15.69:0/2475732137 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b580343f0 con 0x7f6b60142710 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b5effd640 10 monclient: dump: 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:06.136 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:06.137 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:06.137 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:06.137 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:06.137 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:06.137 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:06.137 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:06.137 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:06.137 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b66621640 5 monclient: authenticate success, global_id 4190 2024-01-21T04:33:06.137 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b66621640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:06.137 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.134+0000 7f6b66621640 4 set_mon_vals no callback set 2024-01-21T04:33:06.139 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.135+0000 7f6b66621640 10 monclient: shutdown 2024-01-21T04:33:06.139 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.135+0000 7f6b66621640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:06.139 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.135+0000 7f6b66621640 1 -- 172.21.15.69:0/2475732137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 msgr2=0x7f6b60142b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:06.139 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.135+0000 7f6b66621640 1 --2- 172.21.15.69:0/2475732137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 0x7f6b60142b40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009e50 tx=0x7f6b5802be40 comp rx=0 tx=0).stop 2024-01-21T04:33:06.139 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.135+0000 7f6b66621640 1 -- 172.21.15.69:0/2475732137 shutdown_connections 2024-01-21T04:33:06.139 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.135+0000 7f6b66621640 1 --2- 172.21.15.69:0/2475732137 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 0x7f6b60142b40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:06.139 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.135+0000 7f6b66621640 1 --2- 172.21.15.69:0/2475732137 >> v2:172.21.15.144:3300/0 conn(0x7f6b6013fa10 0x7f6b601421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:06.139 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.135+0000 7f6b66621640 1 -- 172.21.15.69:0/2475732137 >> 172.21.15.69:0/2475732137 conn(0x7f6b600670b0 msgr2=0x7f6b600b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.135+0000 7f6b66621640 1 -- 172.21.15.69:0/2475732137 shutdown_connections 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.135+0000 7f6b66621640 1 -- 172.21.15.69:0/2475732137 wait complete. 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.136+0000 7f6b66621640 10 monclient: build_initial_monmap 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.136+0000 7f6b66621640 10 monclient: monmap: 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:06.140 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:06.141 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.137+0000 7f6b66621640 1 librados: starting msgr at 2024-01-21T04:33:06.141 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.137+0000 7f6b66621640 1 librados: starting objecter 2024-01-21T04:33:06.141 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.137+0000 7f6b66621640 1 Processor -- start 2024-01-21T04:33:06.141 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.137+0000 7f6b66621640 1 -- start start 2024-01-21T04:33:06.141 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.137+0000 7f6b66621640 1 librados: setting wanted keys 2024-01-21T04:33:06.141 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.138+0000 7f6b66621640 1 librados: calling monclient init 2024-01-21T04:33:06.141 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.138+0000 7f6b66621640 10 monclient: init 2024-01-21T04:33:06.145 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b66621640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:06.145 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b66621640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:33:06.145 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b66621640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b6013fa10 0x7f6b601db000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:06.145 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b66621640 10 monclient(hunting): picked mon.noname-a con 0x7f6b6013fa10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:06.145 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b66621640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 0x7f6b60146c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:06.145 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b66621640 10 monclient(hunting): picked mon.noname-c con 0x7f6b60142710 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:06.145 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b66621640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b601dba40 0x7f6b601471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b66621640 10 monclient(hunting): picked mon.noname-b con 0x7f6b601dba40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b66621640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b66621640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b601487f0 con 0x7f6b6013fa10 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b66621640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b6013fa10 0x7f6b601db000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.139+0000 7f6b5f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 0x7f6b60146c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b66621640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b60148ad0 con 0x7f6b601dba40 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b66621640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b66621640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b601461d0 con 0x7f6b60142710 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b66621640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b5ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b6013fa10 0x7f6b601db000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55264/0 (socket says 172.21.15.69:55264) 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b5ffff640 1 -- 172.21.15.69:0/3700587414 learned_addr learned my addr 172.21.15.69:0/3700587414 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b66621640 10 monclient(hunting): authenticate will time out at 1344.481569s 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b64bbe640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b601dba40 0x7f6b601471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:06.146 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b5ffff640 10 monclient(hunting): get_auth_request con 0x7f6b6013fa10 auth_method 0 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b5ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b5f7fe640 10 monclient(hunting): get_auth_request con 0x7f6b60142710 auth_method 0 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b5f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b64bbe640 10 monclient(hunting): get_auth_request con 0x7f6b601dba40 auth_method 0 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b64bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b64bbe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.140+0000 7f6b64bbe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.141+0000 7f6b5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.141+0000 7f6b5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.141+0000 7f6b5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:06.147 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.141+0000 7f6b5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:06.148 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.141+0000 7f6b5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:06.148 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.141+0000 7f6b5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:06.148 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.141+0000 7f6b64bbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:06.148 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.141+0000 7f6b64bbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:06.148 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.141+0000 7f6b64bbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:06.149 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.145+0000 7f6b5f7fe640 10 monclient(hunting): handle_auth_done global_id 4193 payload 824 2024-01-21T04:33:06.149 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.145+0000 7f6b5f7fe640 1 -- 172.21.15.69:0/3700587414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b601dba40 msgr2=0x7f6b601471b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:06.149 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.145+0000 7f6b5f7fe640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b601dba40 0x7f6b601471b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:06.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.145+0000 7f6b5f7fe640 1 -- 172.21.15.69:0/3700587414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b6013fa10 msgr2=0x7f6b601db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:06.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.145+0000 7f6b5f7fe640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b6013fa10 0x7f6b601db000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:06.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.145+0000 7f6b5f7fe640 10 monclient: _finish_hunting 0 2024-01-21T04:33:06.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.145+0000 7f6b5f7fe640 1 monclient: found mon.noname-c 2024-01-21T04:33:06.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.145+0000 7f6b5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:06.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.145+0000 7f6b5f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:33:06.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.145+0000 7f6b5f7fe640 1 -- 172.21.15.69:0/3700587414 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b58009b30 con 0x7f6b60142710 2024-01-21T04:33:06.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.145+0000 7f6b5ffff640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b6013fa10 0x7f6b601db000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T04:33:06.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.148+0000 7f6b5f7fe640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 0x7f6b60146c70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6b5400b3f0 tx=0x7f6b5400b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:06.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.148+0000 7f6b5cff9640 10 client.?.objecter ms_handle_connect 0x7f6b60142710 2024-01-21T04:33:06.150 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.148+0000 7f6b5cff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:06.154 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.152+0000 7f6b5cff9640 1 -- 172.21.15.69:0/3700587414 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54011830 con 0x7f6b60142710 2024-01-21T04:33:06.154 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.152+0000 7f6b5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:06.154 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.152+0000 7f6b5cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:06.154 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.152+0000 7f6b5cff9640 10 monclient: dump: 2024-01-21T04:33:06.154 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:06.154 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:06.154 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:06.154 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:06.154 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.152+0000 7f6b5cff9640 10 monclient: _finish_auth 0 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.152+0000 7f6b5cff9640 10 monclient: _check_auth_tickets 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.152+0000 7f6b5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b5cff9640 1 -- 172.21.15.69:0/3700587414 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b54011990 con 0x7f6b60142710 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b5cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b5cff9640 1 -- 172.21.15.69:0/3700587414 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54011d40 con 0x7f6b60142710 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b5cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b5cff9640 10 monclient: dump: 2024-01-21T04:33:06.155 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b5e7fc640 4 set_mon_vals no callback set 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b66621640 5 monclient: authenticate success, global_id 4193 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b66621640 10 monclient: _renew_subs 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b66621640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:06.156 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b66621640 1 -- 172.21.15.69:0/3700587414 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b601476f0 con 0x7f6b60142710 2024-01-21T04:33:06.157 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b66621640 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:06.157 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b66621640 10 monclient: _renew_subs 2024-01-21T04:33:06.157 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b66621640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:06.157 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b66621640 1 -- 172.21.15.69:0/3700587414 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b601d7a70 con 0x7f6b60142710 2024-01-21T04:33:06.157 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.153+0000 7f6b66621640 1 librados: init done 2024-01-21T04:33:06.157 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.154+0000 7f6b66621640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:06.158 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.154+0000 7f6b66621640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:06.158 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.154+0000 7f6b66621640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:06.158 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.154+0000 7f6b66621640 1 -- 172.21.15.69:0/3700587414 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b60075c30 con 0x7f6b60142710 2024-01-21T04:33:06.185 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.184+0000 7f6b5cff9640 1 -- 172.21.15.69:0/3700587414 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7f6b5401a400 con 0x7f6b60142710 2024-01-21T04:33:06.185 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.184+0000 7f6b5cff9640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b4c06c820 0x7f6b4c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:06.185 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.185+0000 7f6b5ffff640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b4c06c820 0x7f6b4c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:06.186 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.185+0000 7f6b5ffff640 10 monclient: get_auth_request con 0x7f6b4c06c820 auth_method 0 2024-01-21T04:33:06.186 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.185+0000 7f6b5ffff640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b4c06c820 0x7f6b4c06ece0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6b50001c80 tx=0x7f6b50009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:06.186 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.185+0000 7f6b5cff9640 10 client.4193.objecter ms_handle_connect 0x7f6b4c06c820 2024-01-21T04:33:06.190 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.189+0000 7f6b5cff9640 1 -- 172.21.15.69:0/3700587414 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6b54058820 con 0x7f6b60142710 2024-01-21T04:33:06.190 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.189+0000 7f6b5cff9640 10 client.4193.objecter ms_dispatch 0x7f6b60000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T04:33:06.190 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.189+0000 7f6b5cff9640 3 client.4193.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T04:33:06.190 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.189+0000 7f6b5cff9640 3 client.4193.objecter handle_osd_map decoding full epoch 10 2024-01-21T04:33:06.190 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.189+0000 7f6b5cff9640 20 client.4193.objecter dump_active .. 0 homeless 2024-01-21T04:33:06.240 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:17.271 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:06.571 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.571+0000 7f6b5cff9640 1 -- 172.21.15.69:0/3700587414 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6b54054b60 con 0x7f6b60142710 2024-01-21T04:33:06.572 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.571+0000 7f6b5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:06.572 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.571+0000 7f6b5cff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:06.941 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.940+0000 7f6b66621640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:06.941 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.940+0000 7f6b66621640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:06.941 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.940+0000 7f6b66621640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:06.941 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.940+0000 7f6b66621640 1 -- 172.21.15.69:0/3700587414 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b60146360 con 0x7f6b60142710 2024-01-21T04:33:06.966 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:18.044 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:06.969 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.968+0000 7f6b5cff9640 1 -- 172.21.15.69:0/3700587414 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f6b54015020 con 0x7f6b60142710 2024-01-21T04:33:06.969 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.968+0000 7f6b5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:06.969 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.968+0000 7f6b5cff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:06.969 INFO:teuthology.misc.health.smithi069.stdout: 2024-01-21T04:33:06.969 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T04:32:48.763226+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T04:33:06.971 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.970+0000 7f6b66621640 10 librados: watch_flush enter 2024-01-21T04:33:06.971 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.970+0000 7f6b66621640 10 librados: watch_flush exit 2024-01-21T04:33:06.971 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.970+0000 7f6b66621640 20 client.4193.objecter shutdown clearing up homeless session... 2024-01-21T04:33:06.971 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.970+0000 7f6b66621640 10 client.4193.objecter successfully canceled tick 2024-01-21T04:33:06.977 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 1 -- 172.21.15.69:0/3700587414 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b4c06c820 msgr2=0x7f6b4c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:06.977 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b4c06c820 0x7f6b4c06ece0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6b50001c80 tx=0x7f6b50009040 comp rx=0 tx=0).stop 2024-01-21T04:33:06.977 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 10 monclient: shutdown 2024-01-21T04:33:06.977 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:06.977 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 1 -- 172.21.15.69:0/3700587414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 msgr2=0x7f6b60146c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:06.977 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 0x7f6b60146c70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6b5400b3f0 tx=0x7f6b5400b8b0 comp rx=0 tx=0).stop 2024-01-21T04:33:06.977 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 1 -- 172.21.15.69:0/3700587414 shutdown_connections 2024-01-21T04:33:06.977 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b4c06c820 0x7f6b4c06ece0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:06.978 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b601dba40 0x7f6b601471b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:06.978 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b60142710 0x7f6b60146c70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:06.978 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 1 --2- 172.21.15.69:0/3700587414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b6013fa10 0x7f6b601db000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:06.978 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.976+0000 7f6b66621640 1 -- 172.21.15.69:0/3700587414 >> 172.21.15.69:0/3700587414 conn(0x7f6b600a8a60 msgr2=0x7f6b6013f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:06.978 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.977+0000 7f6b66621640 1 -- 172.21.15.69:0/3700587414 shutdown_connections 2024-01-21T04:33:06.978 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.977+0000 7f6b66621640 1 -- 172.21.15.69:0/3700587414 wait complete. 2024-01-21T04:33:06.978 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:06.978+0000 7f6b66621640 1 librados: shutdown 2024-01-21T04:33:06.991 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-21T04:33:07.087 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:18.173 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:10.461 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:21.492 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:10.464 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:21.496 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:10.866 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:21.940 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:10.872 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:21.947 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:10.975 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:22.005 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:10.978 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:22.009 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:11.108 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:22.165 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:11.114 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:22.171 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:11.150 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:22.244 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:11.154 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:22.248 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:11.190 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:22.235 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:11.194 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:22.239 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:11.741 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:22.827 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:11.746 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:22.832 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:12.144 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:23.222 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:12.148 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:23.226 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:13.798 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.885+0000 7fc2a6446640 10 monclient: get_monmap_and_config 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.885+0000 7fc2a6446640 10 monclient: build_initial_monmap 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.885+0000 7fc2a6446640 10 monclient: monmap: 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:33:13.886867+0000 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:33:13.886867+0000 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:13.887 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:13.888 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.886+0000 7fc2a6446640 1 Processor -- start 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.886+0000 7fc2a6446640 1 -- start start 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.886+0000 7fc2a6446640 10 monclient: init 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a00a8a60 0x7fc2a013f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 10 monclient(hunting): picked mon.noname-b con 0x7fc2a00a8a60 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2a013f840 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2a01428e0 0x7fc2a014c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 10 monclient(hunting): picked mon.noname-a con 0x7fc2a01428e0 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a00b1de0 con 0x7fc2a013f840 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a00a8e40 con 0x7fc2a01428e0 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:13.891 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00aa480 con 0x7fc2a00a8a60 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.887+0000 7fc2a6446640 10 monclient(hunting): authenticate will time out at 1352.228649s 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc2a49e3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2a01428e0 0x7fc2a014c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc29ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a00a8a60 0x7fc2a013f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc29ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a00a8a60 0x7fc2a013f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37940/0 (socket says 172.21.15.69:37940) 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc29ffff640 1 -- 172.21.15.69:0/374927573 learned_addr learned my addr 172.21.15.69:0/374927573 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc29ffff640 10 monclient(hunting): get_auth_request con 0x7fc2a00a8a60 auth_method 0 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc29ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc29ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc29ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc2a49e3640 10 monclient(hunting): get_auth_request con 0x7fc2a01428e0 auth_method 0 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc2a49e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc2a49e3640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:13.892 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.888+0000 7fc2a49e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:13.893 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.889+0000 7fc2a49e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:13.893 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.889+0000 7fc2a49e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:13.893 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.889+0000 7fc2a49e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:13.893 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.889+0000 7fc29ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:13.893 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.889+0000 7fc29ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:13.893 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.889+0000 7fc29ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:13.894 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.892+0000 7fc2a49e3640 10 monclient(hunting): handle_auth_done global_id 4206 payload 274 2024-01-21T04:33:13.894 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.892+0000 7fc2a49e3640 1 -- 172.21.15.69:0/374927573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a00a8a60 msgr2=0x7fc2a013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:13.894 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.892+0000 7fc2a49e3640 1 --2- 172.21.15.69:0/374927573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a00a8a60 0x7fc2a013f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:13.894 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.892+0000 7fc2a49e3640 1 -- 172.21.15.69:0/374927573 >> v1:172.21.15.144:6789/0 conn(0x7fc2a013f840 legacy=0x7fc2a01421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:13.894 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.892+0000 7fc2a49e3640 10 monclient: _finish_hunting 0 2024-01-21T04:33:13.894 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.892+0000 7fc2a49e3640 1 monclient: found mon.noname-a 2024-01-21T04:33:13.894 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.892+0000 7fc2a49e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:13.894 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.892+0000 7fc2a49e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:13.894 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.892+0000 7fc2a49e3640 1 -- 172.21.15.69:0/374927573 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a013fc40 con 0x7fc2a01428e0 2024-01-21T04:33:13.895 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.894+0000 7fc2a49e3640 1 --2- 172.21.15.69:0/374927573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a01428e0 0x7fc2a014c930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc298009d90 tx=0x7fc29802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690cbf7a63acbce9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:13.905 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 1 -- 172.21.15.69:0/374927573 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29802cb60 con 0x7fc2a01428e0 2024-01-21T04:33:13.905 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 10 monclient: dump: 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 10 monclient: _finish_auth 0 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:13.906 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 1 -- 172.21.15.69:0/374927573 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29802ccc0 con 0x7fc2a01428e0 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 1 -- 172.21.15.69:0/374927573 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298034660 con 0x7fc2a01428e0 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc29effd640 10 monclient: dump: 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:13.907 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:13.908 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc2a6446640 5 monclient: authenticate success, global_id 4206 2024-01-21T04:33:13.908 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc2a6446640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:13.908 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.899+0000 7fc2a6446640 4 set_mon_vals no callback set 2024-01-21T04:33:13.908 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.900+0000 7fc2a6446640 10 monclient: shutdown 2024-01-21T04:33:13.908 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.900+0000 7fc2a6446640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:13.908 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.900+0000 7fc2a6446640 1 -- 172.21.15.69:0/374927573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a01428e0 msgr2=0x7fc2a014c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:13.908 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.900+0000 7fc2a6446640 1 --2- 172.21.15.69:0/374927573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a01428e0 0x7fc2a014c930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc298009d90 tx=0x7fc29802bdb0 comp rx=0 tx=0).stop 2024-01-21T04:33:13.908 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.900+0000 7fc2a6446640 1 -- 172.21.15.69:0/374927573 shutdown_connections 2024-01-21T04:33:13.908 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.900+0000 7fc2a6446640 1 --2- 172.21.15.69:0/374927573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a01428e0 0x7fc2a014c930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:13.908 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.900+0000 7fc2a6446640 1 --2- 172.21.15.69:0/374927573 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a00a8a60 0x7fc2a013f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:13.908 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.900+0000 7fc2a6446640 1 -- 172.21.15.69:0/374927573 >> 172.21.15.69:0/374927573 conn(0x7fc2a00670b0 msgr2=0x7fc2a00b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.900+0000 7fc2a6446640 1 -- 172.21.15.69:0/374927573 shutdown_connections 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.900+0000 7fc2a6446640 1 -- 172.21.15.69:0/374927573 wait complete. 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.901+0000 7fc2a6446640 10 monclient: build_initial_monmap 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.901+0000 7fc2a6446640 10 monclient: monmap: 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:13.909 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.901+0000 7fc2a6446640 1 librados: starting msgr at 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.901+0000 7fc2a6446640 1 librados: starting objecter 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.901+0000 7fc2a6446640 1 Processor -- start 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 1 -- start start 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 1 librados: setting wanted keys 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 1 librados: calling monclient init 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 10 monclient: init 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2a013f840 0x7fc2a00b70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 10 monclient(hunting): picked mon.noname-b con 0x7fc2a013f840 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a01428e0 0x7fc2a013bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 10 monclient(hunting): picked mon.noname-c con 0x7fc2a01428e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a013a7f0 0x7fc2a013c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.902+0000 7fc2a6446640 10 monclient(hunting): picked mon.noname-a con 0x7fc2a013a7f0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:13.910 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a6446640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a6446640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00b6770 con 0x7fc2a013a7f0 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a6446640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a6446640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a013b570 con 0x7fc2a013f840 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a6446640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a6446640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00b6c00 con 0x7fc2a01428e0 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a6446640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a6446640 10 monclient(hunting): authenticate will time out at 1352.244335s 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a01428e0 0x7fc2a013bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a49e3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a013a7f0 0x7fc2a013c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29ffff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2a013f840 0x7fc2a00b70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29f7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a01428e0 0x7fc2a013bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:37946/0 (socket says 172.21.15.69:37946) 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29f7fe640 1 -- 172.21.15.69:0/796470171 learned_addr learned my addr 172.21.15.69:0/796470171 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29f7fe640 10 monclient(hunting): get_auth_request con 0x7fc2a01428e0 auth_method 0 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:13.911 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29ffff640 10 monclient(hunting): get_auth_request con 0x7fc2a013f840 auth_method 0 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc29ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a49e3640 10 monclient(hunting): get_auth_request con 0x7fc2a013a7f0 auth_method 0 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a49e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a49e3640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.903+0000 7fc2a49e3640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.904+0000 7fc29ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.904+0000 7fc29ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.904+0000 7fc29ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.904+0000 7fc29f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.904+0000 7fc29f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:13.912 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.904+0000 7fc29f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:13.913 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.904+0000 7fc2a49e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:13.913 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.904+0000 7fc2a49e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:13.913 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.904+0000 7fc2a49e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:13.916 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.907+0000 7fc29ffff640 10 monclient(hunting): handle_auth_done global_id 4210 payload 824 2024-01-21T04:33:13.916 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.907+0000 7fc29ffff640 1 -- 172.21.15.69:0/796470171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a01428e0 msgr2=0x7fc2a013bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:13.916 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.907+0000 7fc29ffff640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a01428e0 0x7fc2a013bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:13.916 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.907+0000 7fc29ffff640 1 -- 172.21.15.69:0/796470171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a013a7f0 msgr2=0x7fc2a013c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:13.916 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.907+0000 7fc29ffff640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a013a7f0 0x7fc2a013c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:13.916 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.907+0000 7fc29ffff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.907+0000 7fc29ffff640 1 monclient: found mon.noname-b 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.907+0000 7fc29ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.907+0000 7fc29ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.907+0000 7fc29ffff640 1 -- 172.21.15.69:0/796470171 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc298009a70 con 0x7fc2a013f840 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.909+0000 7fc29ffff640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2a013f840 0x7fc2a00b70e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc29000e850 tx=0x7fc29000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.909+0000 7fc29cff9640 10 client.?.objecter ms_handle_connect 0x7fc2a013f840 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.909+0000 7fc29cff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc29cff9640 1 -- 172.21.15.69:0/796470171 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29000c940 con 0x7fc2a013f840 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc29cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc29cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc29cff9640 10 monclient: dump: 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:13.917 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc29cff9640 10 monclient: _finish_auth 0 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc29cff9640 10 monclient: _check_auth_tickets 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc29cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc2a6446640 5 monclient: authenticate success, global_id 4210 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc2a6446640 10 monclient: _renew_subs 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc2a6446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc2a6446640 1 -- 172.21.15.69:0/796470171 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a013b7e0 con 0x7fc2a013f840 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc2a6446640 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc2a6446640 10 monclient: _renew_subs 2024-01-21T04:33:13.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.911+0000 7fc2a6446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.912+0000 7fc2a6446640 1 -- 172.21.15.69:0/796470171 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a01404d0 con 0x7fc2a013f840 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.912+0000 7fc2a6446640 1 librados: init done 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.912+0000 7fc2a6446640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.912+0000 7fc2a6446640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.912+0000 7fc2a6446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.912+0000 7fc2a6446640 1 -- 172.21.15.69:0/796470171 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a0075c30 con 0x7fc2a013f840 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.914+0000 7fc29cff9640 1 -- 172.21.15.69:0/796470171 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29000caa0 con 0x7fc2a013f840 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.914+0000 7fc29cff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.914+0000 7fc29cff9640 1 -- 172.21.15.69:0/796470171 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29000ce10 con 0x7fc2a013f840 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.914+0000 7fc29cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.914+0000 7fc29cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.914+0000 7fc29cff9640 10 monclient: dump: 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:13.919 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:13.920 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:13.920 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:13.920 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:13.920 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:13.920 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:13.920 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:13.920 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:13.920 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:13.914+0000 7fc29e7fc640 4 set_mon_vals no callback set 2024-01-21T04:33:14.240 INFO:tasks.ceph.osd.5.smithi144.stderr:==00:00:00:25.271 34568== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:14.276 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.275+0000 7fc29cff9640 1 -- 172.21.15.69:0/796470171 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7fc290013470 con 0x7fc2a013f840 2024-01-21T04:33:14.276 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.275+0000 7fc29cff9640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc28c06c7d0 0x7fc28c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:14.276 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.276+0000 7fc29f7fe640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc28c06c7d0 0x7fc28c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:14.277 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.276+0000 7fc29f7fe640 10 monclient: get_auth_request con 0x7fc28c06c7d0 auth_method 0 2024-01-21T04:33:14.277 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.276+0000 7fc29cff9640 10 client.4210.objecter ms_handle_connect 0x7fc28c06c7d0 2024-01-21T04:33:14.277 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.276+0000 7fc29f7fe640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc28c06c7d0 0x7fc28c06ec90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0141da0 tx=0x7fc294007ef0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:14.279 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.278+0000 7fc29cff9640 1 -- 172.21.15.69:0/796470171 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fc290058b70 con 0x7fc2a013f840 2024-01-21T04:33:14.279 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.278+0000 7fc29cff9640 10 client.4210.objecter ms_dispatch 0x7fc2a0000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T04:33:14.279 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.278+0000 7fc29cff9640 3 client.4210.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T04:33:14.280 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.278+0000 7fc29cff9640 3 client.4210.objecter handle_osd_map decoding full epoch 10 2024-01-21T04:33:14.280 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.279+0000 7fc29cff9640 20 client.4210.objecter dump_active .. 0 homeless 2024-01-21T04:33:14.290 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.289+0000 7fc29cff9640 1 -- 172.21.15.69:0/796470171 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc290054eb0 con 0x7fc2a013f840 2024-01-21T04:33:14.290 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.289+0000 7fc29cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:14.290 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.289+0000 7fc29cff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:14.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.538+0000 7fc2a6446640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:14.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.538+0000 7fc2a6446640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:14.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.538+0000 7fc2a6446640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:14.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.538+0000 7fc2a6446640 1 -- 172.21.15.69:0/796470171 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2a0068300 con 0x7fc2a013f840 2024-01-21T04:33:14.589 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.588+0000 7fc29cff9640 1 -- 172.21.15.69:0/796470171 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fc290021090 con 0x7fc2a013f840 2024-01-21T04:33:14.589 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.588+0000 7fc29cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:14.589 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.588+0000 7fc29cff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:14.589 INFO:teuthology.misc.health.smithi069.stdout: 2024-01-21T04:33:14.589 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T04:32:48.763226+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T04:33:14.592 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.590+0000 7fc2a6446640 10 librados: watch_flush enter 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.590+0000 7fc2a6446640 10 librados: watch_flush exit 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.590+0000 7fc2a6446640 20 client.4210.objecter shutdown clearing up homeless session... 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.590+0000 7fc2a6446640 10 client.4210.objecter successfully canceled tick 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.590+0000 7fc2a6446640 1 -- 172.21.15.69:0/796470171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc28c06c7d0 msgr2=0x7fc28c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.590+0000 7fc2a6446640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc28c06c7d0 0x7fc28c06ec90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0141da0 tx=0x7fc294007ef0 comp rx=0 tx=0).stop 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.590+0000 7fc2a6446640 10 monclient: shutdown 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.590+0000 7fc2a6446640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.590+0000 7fc2a6446640 1 -- 172.21.15.69:0/796470171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2a013f840 msgr2=0x7fc2a00b70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.590+0000 7fc2a6446640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2a013f840 0x7fc2a00b70e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc29000e850 tx=0x7fc29000ed10 comp rx=0 tx=0).stop 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.591+0000 7fc2a6446640 1 -- 172.21.15.69:0/796470171 shutdown_connections 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.591+0000 7fc2a6446640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc28c06c7d0 0x7fc28c06ec90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.591+0000 7fc2a6446640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2a013a7f0 0x7fc2a013c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.591+0000 7fc2a6446640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2a01428e0 0x7fc2a013bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:14.593 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.591+0000 7fc2a6446640 1 --2- 172.21.15.69:0/796470171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2a013f840 0x7fc2a00b70e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:14.594 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.591+0000 7fc2a6446640 1 -- 172.21.15.69:0/796470171 >> 172.21.15.69:0/796470171 conn(0x7fc2a00a8a60 msgr2=0x7fc2a0073df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:14.594 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.591+0000 7fc2a6446640 1 -- 172.21.15.69:0/796470171 shutdown_connections 2024-01-21T04:33:14.594 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.591+0000 7fc2a6446640 1 -- 172.21.15.69:0/796470171 wait complete. 2024-01-21T04:33:14.594 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:14.591+0000 7fc2a6446640 1 librados: shutdown 2024-01-21T04:33:14.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-21T04:33:15.028 INFO:tasks.ceph.osd.6.smithi144.stderr:==00:00:00:26.059 34569== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:15.240 INFO:tasks.ceph.osd.2.smithi069.stderr:==00:00:00:26.314 34553== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:15.327 INFO:tasks.ceph.osd.1.smithi069.stderr:==00:00:00:26.421 34535== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:15.575 INFO:tasks.ceph.osd.7.smithi144.stderr:==00:00:00:26.620 34567== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:15.692 INFO:tasks.ceph.osd.4.smithi144.stderr:==00:00:00:26.749 34566== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:16.162 INFO:tasks.ceph.osd.3.smithi069.stderr:==00:00:00:27.248 34543== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:16.825 INFO:tasks.ceph.osd.0.smithi069.stderr:==00:00:00:27.903 34552== Warning: unimplemented fcntl command: 1038 2024-01-21T04:33:21.414 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T04:33:21.518 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.517+0000 7f57f3615640 10 monclient: get_monmap_and_config 2024-01-21T04:33:21.518 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.517+0000 7f57f3615640 10 monclient: build_initial_monmap 2024-01-21T04:33:21.518 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.517+0000 7f57f3615640 10 monclient: monmap: 2024-01-21T04:33:21.518 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:21.518 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:21.518 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:33:21.518626+0000 2024-01-21T04:33:21.519 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:33:21.518626+0000 2024-01-21T04:33:21.519 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:21.519 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:21.519 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:21.519 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:21.519 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:21.519 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:21.519 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:21.519 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:21.521 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.518+0000 7f57f3615640 1 Processor -- start 2024-01-21T04:33:21.521 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.518+0000 7f57f3615640 1 -- start start 2024-01-21T04:33:21.521 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.518+0000 7f57f3615640 10 monclient: init 2024-01-21T04:33:21.521 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.518+0000 7f57f3615640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:21.521 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-01-21T04:33:21.521 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f57ec13f740 0x7f57ec0a9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 10 monclient(hunting): picked mon.noname-c con 0x7f57ec13f740 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57ec0b1b10 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f57ec0a9670 0x7f57ec1429e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 10 monclient(hunting): picked mon.noname-a con 0x7f57ec0a9670 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57ec0a8ee0 con 0x7f57ec0b1b10 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ec142f20 con 0x7f57ec0a9670 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57ec0aad70 con 0x7f57ec13f740 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f3615640 10 monclient(hunting): authenticate will time out at 1359.860297s 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f1bb2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f57ec0a9670 0x7f57ec1429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f1bb2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f57ec0a9670 0x7f57ec1429e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46236/0 (socket says 172.21.15.69:46236) 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f1bb2640 1 -- 172.21.15.69:0/2441862424 learned_addr learned my addr 172.21.15.69:0/2441862424 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:21.522 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f1bb2640 10 monclient(hunting): get_auth_request con 0x7f57ec0a9670 auth_method 0 2024-01-21T04:33:21.523 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f1bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:21.523 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f1bb2640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:21.523 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f1bb2640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:21.523 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.519+0000 7f57f13b1640 1 --2- 172.21.15.69:0/2441862424 >> v2:172.21.15.144:3300/0 conn(0x7f57ec13f740 0x7f57ec0a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:21.523 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.520+0000 7f57f1bb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:21.523 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.520+0000 7f57f1bb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:21.523 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.520+0000 7f57f1bb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:21.523 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.520+0000 7f57f13b1640 10 monclient(hunting): get_auth_request con 0x7f57ec13f740 auth_method 0 2024-01-21T04:33:21.523 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.520+0000 7f57f13b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:21.523 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.520+0000 7f57f13b1640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:21.523 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.520+0000 7f57f13b1640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:21.524 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f13b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f13b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f13b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f1bb2640 10 monclient(hunting): handle_auth_done global_id 4212 payload 274 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f1bb2640 1 -- 172.21.15.69:0/2441862424 >> v2:172.21.15.144:3300/0 conn(0x7f57ec13f740 msgr2=0x7f57ec0a9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f1bb2640 1 --2- 172.21.15.69:0/2441862424 >> v2:172.21.15.144:3300/0 conn(0x7f57ec13f740 0x7f57ec0a9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f1bb2640 1 -- 172.21.15.69:0/2441862424 >> v1:172.21.15.144:6789/0 conn(0x7f57ec0b1b10 legacy=0x7f57ec1422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f1bb2640 10 monclient: _finish_hunting 0 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f1bb2640 1 monclient: found mon.noname-a 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f1bb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f1bb2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.522+0000 7f57f1bb2640 1 -- 172.21.15.69:0/2441862424 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec13fb20 con 0x7f57ec0a9670 2024-01-21T04:33:21.525 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.524+0000 7f57f1bb2640 1 --2- 172.21.15.69:0/2441862424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57ec0a9670 0x7f57ec1429e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0b53f0 tx=0x7f57e402bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d432db655e6496 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:21.529 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 1 -- 172.21.15.69:0/2441862424 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e402caa0 con 0x7f57ec0a9670 2024-01-21T04:33:21.529 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:21.529 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:21.529 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 10 monclient: dump: 2024-01-21T04:33:21.529 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:21.529 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:21.529 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:21.529 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:21.529 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:21.529 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 10 monclient: _finish_auth 0 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 10 monclient: _check_auth_tickets 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 1 -- 172.21.15.69:0/2441862424 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e402cc00 con 0x7f57ec0a9670 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 1 -- 172.21.15.69:0/2441862424 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e40343f0 con 0x7f57ec0a9670 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:21.530 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57e3fff640 10 monclient: dump: 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57f3615640 5 monclient: authenticate success, global_id 4212 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57f3615640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:21.531 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.528+0000 7f57f3615640 4 set_mon_vals no callback set 2024-01-21T04:33:21.536 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.529+0000 7f57f3615640 10 monclient: shutdown 2024-01-21T04:33:21.536 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.529+0000 7f57f3615640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:21.536 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.529+0000 7f57f3615640 1 -- 172.21.15.69:0/2441862424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57ec0a9670 msgr2=0x7f57ec1429e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:21.536 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.529+0000 7f57f3615640 1 --2- 172.21.15.69:0/2441862424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57ec0a9670 0x7f57ec1429e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0b53f0 tx=0x7f57e402bf00 comp rx=0 tx=0).stop 2024-01-21T04:33:21.536 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.529+0000 7f57f3615640 1 -- 172.21.15.69:0/2441862424 shutdown_connections 2024-01-21T04:33:21.536 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.529+0000 7f57f3615640 1 --2- 172.21.15.69:0/2441862424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57ec0a9670 0x7f57ec1429e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:21.536 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.529+0000 7f57f3615640 1 --2- 172.21.15.69:0/2441862424 >> v2:172.21.15.144:3300/0 conn(0x7f57ec13f740 0x7f57ec0a9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:21.536 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.529+0000 7f57f3615640 1 -- 172.21.15.69:0/2441862424 >> 172.21.15.69:0/2441862424 conn(0x7f57ec066e50 msgr2=0x7f57ec0b7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.529+0000 7f57f3615640 1 -- 172.21.15.69:0/2441862424 shutdown_connections 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.529+0000 7f57f3615640 1 -- 172.21.15.69:0/2441862424 wait complete. 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.530+0000 7f57f3615640 10 monclient: build_initial_monmap 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.530+0000 7f57f3615640 10 monclient: monmap: 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:21.537 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.530+0000 7f57f3615640 1 librados: starting msgr at 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.530+0000 7f57f3615640 1 librados: starting objecter 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 1 Processor -- start 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 1 -- start start 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 1 librados: setting wanted keys 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 1 librados: calling monclient init 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 10 monclient: init 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57ec0a9670 0x7f57ec13f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 10 monclient(hunting): picked mon.noname-c con 0x7f57ec0a9670 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57ec0b1b10 0x7f57ec0ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 10 monclient(hunting): picked mon.noname-a con 0x7f57ec0b1b10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.531+0000 7f57f3615640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57ec13f740 0x7f57ec138b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:21.538 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.532+0000 7f57f3615640 10 monclient(hunting): picked mon.noname-b con 0x7f57ec13f740 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.532+0000 7f57f3615640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.532+0000 7f57f3615640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec0afdf0 con 0x7f57ec0b1b10 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.532+0000 7f57f3615640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.532+0000 7f57f3615640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec13e6b0 con 0x7f57ec13f740 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.532+0000 7f57f3615640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.532+0000 7f57f3615640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57ec0afa50 con 0x7f57ec0a9670 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.532+0000 7f57f3615640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.532+0000 7f57f3615640 10 monclient(hunting): authenticate will time out at 1359.873444s 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.532+0000 7f57f1bb2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57ec13f740 0x7f57ec138b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.532+0000 7f57f13b1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57ec0a9670 0x7f57ec13f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f13b1640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57ec0a9670 0x7f57ec13f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:33636/0 (socket says 172.21.15.69:33636) 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f13b1640 1 -- 172.21.15.69:0/2333051184 learned_addr learned my addr 172.21.15.69:0/2333051184 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f13b1640 10 monclient(hunting): get_auth_request con 0x7f57ec0a9670 auth_method 0 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f13b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:21.539 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f13b1640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f13b1640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f1bb2640 10 monclient(hunting): get_auth_request con 0x7f57ec13f740 auth_method 0 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f1bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f1bb2640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f1bb2640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f0bb0640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57ec0b1b10 0x7f57ec0ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f0bb0640 10 monclient(hunting): get_auth_request con 0x7f57ec0b1b10 auth_method 0 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f0bb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f0bb0640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f0bb0640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f13b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f13b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f13b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:21.540 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f1bb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:21.541 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f1bb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:21.541 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.533+0000 7f57f1bb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:21.541 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.534+0000 7f57f0bb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:21.541 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.534+0000 7f57f0bb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:21.541 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.534+0000 7f57f0bb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:21.542 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.537+0000 7f57f13b1640 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2024-01-21T04:33:21.542 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.537+0000 7f57f13b1640 1 -- 172.21.15.69:0/2333051184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57ec13f740 msgr2=0x7f57ec138b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:21.543 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.537+0000 7f57f13b1640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57ec13f740 0x7f57ec138b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:21.543 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.537+0000 7f57f13b1640 1 -- 172.21.15.69:0/2333051184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57ec0b1b10 msgr2=0x7f57ec0ab430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:21.543 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.537+0000 7f57f13b1640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57ec0b1b10 0x7f57ec0ab430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:21.543 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.537+0000 7f57f13b1640 10 monclient: _finish_hunting 0 2024-01-21T04:33:21.543 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.537+0000 7f57f13b1640 1 monclient: found mon.noname-c 2024-01-21T04:33:21.543 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.537+0000 7f57f13b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:21.543 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.537+0000 7f57f13b1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:33:21.543 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.537+0000 7f57f13b1640 1 -- 172.21.15.69:0/2333051184 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e4009b30 con 0x7f57ec0a9670 2024-01-21T04:33:21.543 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.541+0000 7f57f13b1640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57ec0a9670 0x7f57ec13f120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f57e800e9b0 tx=0x7f57e800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:21.543 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.541+0000 7f57e1ffb640 10 client.?.objecter ms_handle_connect 0x7f57ec0a9670 2024-01-21T04:33:21.543 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.541+0000 7f57e1ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:21.546 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.544+0000 7f57e1ffb640 1 -- 172.21.15.69:0/2333051184 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e8009670 con 0x7f57ec0a9670 2024-01-21T04:33:21.546 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.544+0000 7f57e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.544+0000 7f57e1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.544+0000 7f57e1ffb640 10 monclient: dump: 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.544+0000 7f57e1ffb640 10 monclient: _finish_auth 0 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.544+0000 7f57e1ffb640 10 monclient: _check_auth_tickets 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.544+0000 7f57e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:21.547 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57f3615640 5 monclient: authenticate success, global_id 4202 2024-01-21T04:33:21.548 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57f3615640 10 monclient: _renew_subs 2024-01-21T04:33:21.548 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57f3615640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:21.548 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57f3615640 1 -- 172.21.15.69:0/2333051184 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57ec139060 con 0x7f57ec0a9670 2024-01-21T04:33:21.548 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57e1ffb640 1 -- 172.21.15.69:0/2333051184 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e800c9a0 con 0x7f57ec0a9670 2024-01-21T04:33:21.548 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57f3615640 10 client.4202.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:21.548 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57f3615640 10 monclient: _renew_subs 2024-01-21T04:33:21.548 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57f3615640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:21.548 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57f3615640 1 -- 172.21.15.69:0/2333051184 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57ec139540 con 0x7f57ec0a9670 2024-01-21T04:33:21.548 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57f3615640 1 librados: init done 2024-01-21T04:33:21.548 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:21.549 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.545+0000 7f57e37fe640 4 set_mon_vals no callback set 2024-01-21T04:33:21.549 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.546+0000 7f57e1ffb640 1 -- 172.21.15.69:0/2333051184 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e800cd50 con 0x7f57ec0a9670 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.546+0000 7f57e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.546+0000 7f57e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.546+0000 7f57e1ffb640 10 monclient: dump: 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:21.550 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.546+0000 7f57f3615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:21.551 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.546+0000 7f57f3615640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:21.551 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.546+0000 7f57f3615640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:21.551 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.546+0000 7f57f3615640 1 -- 172.21.15.69:0/2333051184 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57cc005080 con 0x7f57ec0a9670 2024-01-21T04:33:21.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.917+0000 7f57e1ffb640 1 -- 172.21.15.69:0/2333051184 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7f57e8013470 con 0x7f57ec0a9670 2024-01-21T04:33:21.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.917+0000 7f57e1ffb640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f57d006c820 0x7f57d006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:21.918 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.918+0000 7f57f0bb0640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f57d006c820 0x7f57d006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:21.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.918+0000 7f57f0bb0640 10 monclient: get_auth_request con 0x7f57d006c820 auth_method 0 2024-01-21T04:33:21.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.918+0000 7f57e1ffb640 10 client.4202.objecter ms_handle_connect 0x7f57d006c820 2024-01-21T04:33:21.919 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.918+0000 7f57f0bb0640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f57d006c820 0x7f57d006ece0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f57ec1d4e60 tx=0x7f57dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:21.934 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.933+0000 7f57e1ffb640 1 -- 172.21.15.69:0/2333051184 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f57e8058950 con 0x7f57ec0a9670 2024-01-21T04:33:21.934 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.933+0000 7f57e1ffb640 10 client.4202.objecter ms_dispatch 0x7f57ec000b90 osd_map(10..10 src has 1..10) v4 2024-01-21T04:33:21.934 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.933+0000 7f57e1ffb640 3 client.4202.objecter handle_osd_map got epochs [10,10] > 0 2024-01-21T04:33:21.934 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.933+0000 7f57e1ffb640 3 client.4202.objecter handle_osd_map decoding full epoch 10 2024-01-21T04:33:21.934 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.933+0000 7f57e1ffb640 20 client.4202.objecter dump_active .. 0 homeless 2024-01-21T04:33:21.935 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.933+0000 7f57e1ffb640 1 -- 172.21.15.69:0/2333051184 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f57e8054c90 con 0x7f57ec0a9670 2024-01-21T04:33:21.935 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.933+0000 7f57e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:21.935 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:21.933+0000 7f57e1ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:22.282 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.281+0000 7f57f3615640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:22.283 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.282+0000 7f57f3615640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:22.283 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.282+0000 7f57f3615640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:22.283 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.282+0000 7f57f3615640 1 -- 172.21.15.69:0/2333051184 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57cc005600 con 0x7f57ec0a9670 2024-01-21T04:33:22.335 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.334+0000 7f57e1ffb640 1 -- 172.21.15.69:0/2333051184 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f57e8021090 con 0x7f57ec0a9670 2024-01-21T04:33:22.336 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.334+0000 7f57e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:22.336 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.334+0000 7f57e1ffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:22.336 INFO:teuthology.misc.health.smithi069.stdout: 2024-01-21T04:33:22.336 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T04:32:48.763226+0000","last_up_change":"0.000000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T04:33:22.339 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 10 librados: watch_flush enter 2024-01-21T04:33:22.339 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 10 librados: watch_flush exit 2024-01-21T04:33:22.339 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 20 client.4202.objecter shutdown clearing up homeless session... 2024-01-21T04:33:22.339 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 10 client.4202.objecter successfully canceled tick 2024-01-21T04:33:22.339 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 1 -- 172.21.15.69:0/2333051184 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f57d006c820 msgr2=0x7f57d006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:22.339 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f57d006c820 0x7f57d006ece0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f57ec1d4e60 tx=0x7f57dc009040 comp rx=0 tx=0).stop 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 10 monclient: shutdown 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 1 -- 172.21.15.69:0/2333051184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57ec0a9670 msgr2=0x7f57ec13f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57ec0a9670 0x7f57ec13f120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f57e800e9b0 tx=0x7f57e800ee70 comp rx=0 tx=0).stop 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 1 -- 172.21.15.69:0/2333051184 shutdown_connections 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f57d006c820 0x7f57d006ece0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57ec13f740 0x7f57ec138b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57ec0b1b10 0x7f57ec0ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 1 --2- 172.21.15.69:0/2333051184 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f57ec0a9670 0x7f57ec13f120 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 1 -- 172.21.15.69:0/2333051184 >> 172.21.15.69:0/2333051184 conn(0x7f57ec066e50 msgr2=0x7f57ec14bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.337+0000 7f57f3615640 1 -- 172.21.15.69:0/2333051184 shutdown_connections 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.338+0000 7f57f3615640 1 -- 172.21.15.69:0/2333051184 wait complete. 2024-01-21T04:33:22.340 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:22.338+0000 7f57f3615640 1 librados: shutdown 2024-01-21T04:33:22.349 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-21T04:33:22.535 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T04:33:22.532+0000 5dc2300 -1 osd.5 0 log_to_monitors true 2024-01-21T04:33:23.329 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:23.325+0000 5dc2300 -1 osd.6 0 log_to_monitors true 2024-01-21T04:33:23.736 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:33:23.731+0000 5dc2300 -1 osd.1 0 log_to_monitors true 2024-01-21T04:33:24.057 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:33:24.007+0000 5dc2300 -1 osd.2 0 log_to_monitors true 2024-01-21T04:33:24.058 INFO:tasks.ceph.osd.7.smithi144.stderr:2024-01-21T04:33:24.047+0000 5dc2300 -1 osd.7 0 log_to_monitors true 2024-01-21T04:33:24.192 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:24.186+0000 5dc2300 -1 osd.4 0 log_to_monitors true 2024-01-21T04:33:24.582 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:33:24.578+0000 5dc2300 -1 osd.3 0 log_to_monitors true 2024-01-21T04:33:25.415 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:33:25.410+0000 5dc2300 -1 osd.0 0 log_to_monitors true 2024-01-21T04:33:25.789 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T04:33:25.784+0000 c493640 -1 osd.5 0 waiting for initial osdmap 2024-01-21T04:33:26.124 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T04:33:26.109+0000 10252640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T04:33:27.129 INFO:tasks.ceph.osd.7.smithi144.stderr:2024-01-21T04:33:27.126+0000 ac7d640 -1 osd.7 0 waiting for initial osdmap 2024-01-21T04:33:27.138 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:33:27.133+0000 ac7d640 -1 osd.2 0 waiting for initial osdmap 2024-01-21T04:33:27.160 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:33:27.156+0000 c493640 -1 osd.1 0 waiting for initial osdmap 2024-01-21T04:33:27.548 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:33:27.534+0000 10252640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T04:33:27.559 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:33:27.544+0000 10252640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T04:33:27.578 INFO:tasks.ceph.osd.7.smithi144.stderr:2024-01-21T04:33:27.562+0000 10252640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T04:33:28.242 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:33:28.233+0000 c493640 -1 osd.0 0 waiting for initial osdmap 2024-01-21T04:33:28.666 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:33:28.650+0000 10252640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T04:33:29.156 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T04:33:29.249 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.247+0000 7f0e46b3e640 10 monclient: get_monmap_and_config 2024-01-21T04:33:29.249 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.247+0000 7f0e46b3e640 10 monclient: build_initial_monmap 2024-01-21T04:33:29.249 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.247+0000 7f0e46b3e640 10 monclient: monmap: 2024-01-21T04:33:29.249 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:29.249 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:29.249 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:33:29.248423+0000 2024-01-21T04:33:29.249 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:33:29.248423+0000 2024-01-21T04:33:29.249 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:29.250 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:29.250 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:29.250 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:29.250 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:29.250 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:29.250 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:29.250 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:29.250 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 1 Processor -- start 2024-01-21T04:33:29.250 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 1 -- start start 2024-01-21T04:33:29.250 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 10 monclient: init 2024-01-21T04:33:29.253 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:29.253 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-01-21T04:33:29.253 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e400a8a60 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:29.254 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0e4013fa10 0x7f0e401421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:29.254 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 10 monclient(hunting): picked mon.noname-a con 0x7f0e4013fa10 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:29.254 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f0e40142710 0x7f0e40142b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:29.254 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0e40142710 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:29.254 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e400b1de0 con 0x7f0e400a8a60 2024-01-21T04:33:29.254 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:29.254 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e400a8e40 con 0x7f0e4013fa10 2024-01-21T04:33:29.254 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:29.254 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e400aa480 con 0x7f0e40142710 2024-01-21T04:33:29.254 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.248+0000 7f0e46b3e640 10 monclient(hunting): authenticate will time out at 1367.590132s 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.250+0000 7f0e3ffff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0e4013fa10 0x7f0e401421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.250+0000 7f0e448da640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0e400a8a60 0x7f0e4013f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58028/0 (socket says 172.21.15.69:58028) 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.250+0000 7f0e448da640 1 -- 172.21.15.69:0/4005686429 learned_addr learned my addr 172.21.15.69:0/4005686429 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e450db640 1 --2- 172.21.15.69:0/4005686429 >> v2:172.21.15.144:3300/0 conn(0x7f0e40142710 0x7f0e40142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e3ffff640 10 monclient(hunting): get_auth_request con 0x7f0e4013fa10 auth_method 0 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e3ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e450db640 10 monclient(hunting): get_auth_request con 0x7f0e40142710 auth_method 0 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e450db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:29.255 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e450db640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:29.256 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e450db640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:29.256 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:29.256 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:29.256 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.251+0000 7f0e3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:29.256 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.252+0000 7f0e450db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:29.256 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.252+0000 7f0e450db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:29.256 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.252+0000 7f0e450db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:29.257 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.254+0000 7f0e450db640 10 monclient(hunting): handle_auth_done global_id 4246 payload 274 2024-01-21T04:33:29.257 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.254+0000 7f0e450db640 1 -- 172.21.15.69:0/4005686429 >> v2:172.21.15.69:3300/0 conn(0x7f0e4013fa10 msgr2=0x7f0e401421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:29.257 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.254+0000 7f0e450db640 1 --2- 172.21.15.69:0/4005686429 >> v2:172.21.15.69:3300/0 conn(0x7f0e4013fa10 0x7f0e401421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:29.257 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.254+0000 7f0e450db640 1 -- 172.21.15.69:0/4005686429 >> v1:172.21.15.69:6789/0 conn(0x7f0e400a8a60 legacy=0x7f0e4013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:29.257 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.254+0000 7f0e450db640 10 monclient: _finish_hunting 0 2024-01-21T04:33:29.257 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.254+0000 7f0e450db640 1 monclient: found mon.noname-c 2024-01-21T04:33:29.258 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.254+0000 7f0e450db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:29.258 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.254+0000 7f0e450db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:29.258 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.254+0000 7f0e450db640 1 -- 172.21.15.69:0/4005686429 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e400aa6f0 con 0x7f0e40142710 2024-01-21T04:33:29.258 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.256+0000 7f0e450db640 1 --2- 172.21.15.69:0/4005686429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e40142710 0x7f0e40142b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0e400b1bb0 tx=0x7f0e3802be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bac7d264be6f4cfa server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:29.260 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.259+0000 7f0e3f7fe640 1 -- 172.21.15.69:0/4005686429 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3802cbe0 con 0x7f0e40142710 2024-01-21T04:33:29.261 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.259+0000 7f0e3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:29.261 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.259+0000 7f0e3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:29.261 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.259+0000 7f0e3f7fe640 10 monclient: dump: 2024-01-21T04:33:29.261 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:29.261 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:29.261 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:29.261 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:29.261 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:29.261 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:29.261 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:29.261 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.259+0000 7f0e3f7fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.259+0000 7f0e3f7fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.259+0000 7f0e3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.259+0000 7f0e46b3e640 5 monclient: authenticate success, global_id 4246 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.259+0000 7f0e46b3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.260+0000 7f0e3f7fe640 1 -- 172.21.15.69:0/4005686429 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e3802cd40 con 0x7f0e40142710 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.260+0000 7f0e3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.260+0000 7f0e3f7fe640 1 -- 172.21.15.69:0/4005686429 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e38034660 con 0x7f0e40142710 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.260+0000 7f0e46b3e640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:29.262 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.260+0000 7f0e46b3e640 4 set_mon_vals no callback set 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.260+0000 7f0e3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.260+0000 7f0e3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.260+0000 7f0e3f7fe640 10 monclient: dump: 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:29.263 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:29.270 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.261+0000 7f0e46b3e640 10 monclient: shutdown 2024-01-21T04:33:29.270 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.261+0000 7f0e46b3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:29.270 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.261+0000 7f0e46b3e640 1 -- 172.21.15.69:0/4005686429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e40142710 msgr2=0x7f0e40142b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:29.270 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.261+0000 7f0e46b3e640 1 --2- 172.21.15.69:0/4005686429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e40142710 0x7f0e40142b20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0e400b1bb0 tx=0x7f0e3802be30 comp rx=0 tx=0).stop 2024-01-21T04:33:29.270 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.261+0000 7f0e46b3e640 1 -- 172.21.15.69:0/4005686429 shutdown_connections 2024-01-21T04:33:29.270 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.261+0000 7f0e46b3e640 1 --2- 172.21.15.69:0/4005686429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e40142710 0x7f0e40142b20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:29.270 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.261+0000 7f0e46b3e640 1 --2- 172.21.15.69:0/4005686429 >> v2:172.21.15.69:3300/0 conn(0x7f0e4013fa10 0x7f0e401421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:29.270 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.261+0000 7f0e46b3e640 1 -- 172.21.15.69:0/4005686429 >> 172.21.15.69:0/4005686429 conn(0x7f0e400670b0 msgr2=0x7f0e400b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:29.270 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.261+0000 7f0e46b3e640 1 -- 172.21.15.69:0/4005686429 shutdown_connections 2024-01-21T04:33:29.270 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.261+0000 7f0e46b3e640 1 -- 172.21.15.69:0/4005686429 wait complete. 2024-01-21T04:33:29.271 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.262+0000 7f0e46b3e640 10 monclient: build_initial_monmap 2024-01-21T04:33:29.271 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.262+0000 7f0e46b3e640 10 monclient: monmap: 2024-01-21T04:33:29.271 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:29.271 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:29.271 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:29.271 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-01-21T04:33:29.271 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:29.271 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:29.271 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:29.271 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:29.271 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.262+0000 7f0e46b3e640 1 librados: starting msgr at 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.262+0000 7f0e46b3e640 1 librados: starting objecter 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.263+0000 7f0e46b3e640 1 Processor -- start 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.263+0000 7f0e46b3e640 1 -- start start 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.263+0000 7f0e46b3e640 1 librados: setting wanted keys 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.263+0000 7f0e46b3e640 1 librados: calling monclient init 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.263+0000 7f0e46b3e640 10 monclient: init 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4013fa10 0x7f0e40140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:29.272 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 10 monclient(hunting): picked mon.noname-b con 0x7f0e4013fa10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e40142710 0x7f0e40140480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0e40142710 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e401d7ee0 0x7f0e401d8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 10 monclient(hunting): picked mon.noname-a con 0x7f0e401d7ee0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e401d7a90 con 0x7f0e401d7ee0 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4013aa10 con 0x7f0e4013fa10 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e401d8880 con 0x7f0e40142710 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e448da640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4013fa10 0x7f0e40140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:29.273 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.264+0000 7f0e46b3e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e3ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e40142710 0x7f0e40140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e450db640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e401d7ee0 0x7f0e401d8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e46b3e640 10 monclient(hunting): authenticate will time out at 1367.606338s 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e448da640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4013fa10 0x7f0e40140e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:40144/0 (socket says 172.21.15.69:40144) 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e448da640 1 -- 172.21.15.69:0/3237503055 learned_addr learned my addr 172.21.15.69:0/3237503055 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e448da640 10 monclient(hunting): get_auth_request con 0x7f0e4013fa10 auth_method 0 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e448da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e448da640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e448da640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e3ffff640 10 monclient(hunting): get_auth_request con 0x7f0e40142710 auth_method 0 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e3ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:29.274 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e450db640 10 monclient(hunting): get_auth_request con 0x7f0e401d7ee0 auth_method 0 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e450db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e450db640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e450db640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e448da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e448da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.265+0000 7f0e448da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.266+0000 7f0e3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.266+0000 7f0e3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.266+0000 7f0e3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.266+0000 7f0e450db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.266+0000 7f0e450db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:29.275 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.266+0000 7f0e450db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:29.277 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.269+0000 7f0e448da640 10 monclient(hunting): handle_auth_done global_id 4249 payload 824 2024-01-21T04:33:29.277 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.269+0000 7f0e448da640 1 -- 172.21.15.69:0/3237503055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e40142710 msgr2=0x7f0e40140480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:29.277 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.269+0000 7f0e448da640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e40142710 0x7f0e40140480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:29.277 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.269+0000 7f0e448da640 1 -- 172.21.15.69:0/3237503055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e401d7ee0 msgr2=0x7f0e401d8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:29.277 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.269+0000 7f0e448da640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e401d7ee0 0x7f0e401d8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:29.277 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.269+0000 7f0e448da640 10 monclient: _finish_hunting 0 2024-01-21T04:33:29.277 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.269+0000 7f0e448da640 1 monclient: found mon.noname-b 2024-01-21T04:33:29.278 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.269+0000 7f0e448da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:29.278 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.269+0000 7f0e448da640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:29.278 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.269+0000 7f0e448da640 1 -- 172.21.15.69:0/3237503055 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e38009a70 con 0x7f0e4013fa10 2024-01-21T04:33:29.278 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.272+0000 7f0e448da640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4013fa10 0x7f0e40140e90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400c3c0 tx=0x7f0e3400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:29.278 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.273+0000 7f0e3d7fa640 10 client.?.objecter ms_handle_connect 0x7f0e4013fa10 2024-01-21T04:33:29.278 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.273+0000 7f0e3d7fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:29.281 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 1 -- 172.21.15.69:0/3237503055 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34005ce0 con 0x7f0e4013fa10 2024-01-21T04:33:29.281 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:29.281 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:29.281 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 10 monclient: dump: 2024-01-21T04:33:29.281 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:29.281 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 10 monclient: _finish_auth 0 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 10 monclient: _check_auth_tickets 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 1 -- 172.21.15.69:0/3237503055 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e34005e40 con 0x7f0e4013fa10 2024-01-21T04:33:29.282 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 1 -- 172.21.15.69:0/3237503055 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e340072d0 con 0x7f0e4013fa10 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3d7fa640 10 monclient: dump: 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:29.283 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e46b3e640 5 monclient: authenticate success, global_id 4249 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e3effd640 4 set_mon_vals no callback set 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e46b3e640 10 monclient: _renew_subs 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e46b3e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e46b3e640 1 -- 172.21.15.69:0/3237503055 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4013b110 con 0x7f0e4013fa10 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e46b3e640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e46b3e640 10 monclient: _renew_subs 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e46b3e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e46b3e640 1 -- 172.21.15.69:0/3237503055 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e4012fa90 con 0x7f0e4013fa10 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.278+0000 7f0e46b3e640 1 librados: init done 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.279+0000 7f0e46b3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.279+0000 7f0e46b3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:29.284 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.279+0000 7f0e46b3e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:29.285 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.279+0000 7f0e46b3e640 1 -- 172.21.15.69:0/3237503055 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e401409c0 con 0x7f0e4013fa10 2024-01-21T04:33:29.572 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.571+0000 7f0e3d7fa640 1 -- 172.21.15.69:0/3237503055 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7f0e340074f0 con 0x7f0e4013fa10 2024-01-21T04:33:29.572 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.572+0000 7f0e3d7fa640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f0e2406c820 0x7f0e2406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:29.573 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.572+0000 7f0e3ffff640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f0e2406c820 0x7f0e2406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:29.573 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.572+0000 7f0e3ffff640 10 monclient: get_auth_request con 0x7f0e2406c820 auth_method 0 2024-01-21T04:33:29.573 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.572+0000 7f0e3ffff640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f0e2406c820 0x7f0e2406ece0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0e40132a60 tx=0x7f0e2c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:29.573 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.573+0000 7f0e3d7fa640 10 client.4249.objecter ms_handle_connect 0x7f0e2406c820 2024-01-21T04:33:29.575 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.574+0000 7f0e3d7fa640 1 -- 172.21.15.69:0/3237503055 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3631+0+0 (secure 0 0 0) 0x7f0e34057bd0 con 0x7f0e4013fa10 2024-01-21T04:33:29.575 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.574+0000 7f0e3d7fa640 10 client.4249.objecter ms_dispatch 0x7f0e40000b90 osd_map(15..15 src has 1..15) v4 2024-01-21T04:33:29.575 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.574+0000 7f0e3d7fa640 3 client.4249.objecter handle_osd_map got epochs [15,15] > 0 2024-01-21T04:33:29.575 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.574+0000 7f0e3d7fa640 3 client.4249.objecter handle_osd_map decoding full epoch 15 2024-01-21T04:33:29.576 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.574+0000 7f0e3d7fa640 20 client.4249.objecter dump_active .. 0 homeless 2024-01-21T04:33:29.588 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.585+0000 7f0e3d7fa640 1 -- 172.21.15.69:0/3237503055 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f0e34053f10 con 0x7f0e4013fa10 2024-01-21T04:33:29.588 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.585+0000 7f0e3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:29.588 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.585+0000 7f0e3d7fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:29.774 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.773+0000 7f0e46b3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:29.774 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.773+0000 7f0e46b3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:29.774 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.773+0000 7f0e46b3e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:29.775 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.773+0000 7f0e46b3e640 1 -- 172.21.15.69:0/3237503055 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e4013c890 con 0x7f0e4013fa10 2024-01-21T04:33:29.803 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.802+0000 7f0e3d7fa640 1 -- 172.21.15.69:0/3237503055 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+9462 (secure 0 0 0) 0x7f0e34017030 con 0x7f0e4013fa10 2024-01-21T04:33:29.803 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.802+0000 7f0e3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:29.803 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.802+0000 7f0e3d7fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:29.804 INFO:teuthology.misc.health.smithi069.stdout: 2024-01-21T04:33:29.804 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":15,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T04:33:28.929791+0000","last_up_change":"2024-01-21T04:33:28.929791+0000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6825","nonce":335500148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6827","nonce":335500148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6831","nonce":335500148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6829","nonce":335500148}]},"public_addr":"172.21.15.69:6825/335500148","cluster_addr":"172.21.15.69:6827/335500148","heartbeat_back_addr":"172.21.15.69:6831/335500148","heartbeat_front_addr":"172.21.15.69:6829/335500148","state":["exists","up"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6809","nonce":3868480563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6811","nonce":3868480563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6815","nonce":3868480563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6813","nonce":3868480563}]},"public_addr":"172.21.15.69:6809/3868480563","cluster_addr":"172.21.15.69:6811/3868480563","heartbeat_back_addr":"172.21.15.69:6815/3868480563","heartbeat_front_addr":"172.21.15.69:6813/3868480563","state":["exists","up"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6817","nonce":3954012353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6819","nonce":3954012353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6823","nonce":3954012353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6821","nonce":3954012353}]},"public_addr":"172.21.15.69:6817/3954012353","cluster_addr":"172.21.15.69:6819/3954012353","heartbeat_back_addr":"172.21.15.69:6823/3954012353","heartbeat_front_addr":"172.21.15.69:6821/3954012353","state":["exists","up"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6803","nonce":1782989653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6805","nonce":1782989653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6809","nonce":1782989653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6807","nonce":1782989653}]},"public_addr":"172.21.15.144:6803/1782989653","cluster_addr":"172.21.15.144:6805/1782989653","heartbeat_back_addr":"172.21.15.144:6809/1782989653","heartbeat_front_addr":"172.21.15.144:6807/1782989653","state":["exists","up"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6811","nonce":14156089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6815","nonce":14156089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6823","nonce":14156089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6819","nonce":14156089}]},"public_addr":"172.21.15.144:6811/14156089","cluster_addr":"172.21.15.144:6815/14156089","heartbeat_back_addr":"172.21.15.144:6823/14156089","heartbeat_front_addr":"172.21.15.144:6819/14156089","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:24.415555+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T04:33:29.806 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.804+0000 7f0e46b3e640 10 librados: watch_flush enter 2024-01-21T04:33:29.806 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.804+0000 7f0e46b3e640 10 librados: watch_flush exit 2024-01-21T04:33:29.806 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.804+0000 7f0e46b3e640 20 client.4249.objecter shutdown clearing up homeless session... 2024-01-21T04:33:29.806 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.804+0000 7f0e46b3e640 10 client.4249.objecter successfully canceled tick 2024-01-21T04:33:29.806 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.804+0000 7f0e46b3e640 1 -- 172.21.15.69:0/3237503055 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f0e2406c820 msgr2=0x7f0e2406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.804+0000 7f0e46b3e640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f0e2406c820 0x7f0e2406ece0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0e40132a60 tx=0x7f0e2c009190 comp rx=0 tx=0).stop 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.804+0000 7f0e46b3e640 10 monclient: shutdown 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.804+0000 7f0e46b3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.804+0000 7f0e46b3e640 1 -- 172.21.15.69:0/3237503055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4013fa10 msgr2=0x7f0e40140e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.804+0000 7f0e46b3e640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4013fa10 0x7f0e40140e90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400c3c0 tx=0x7f0e3400c880 comp rx=0 tx=0).stop 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.805+0000 7f0e46b3e640 1 -- 172.21.15.69:0/3237503055 shutdown_connections 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.805+0000 7f0e46b3e640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f0e2406c820 0x7f0e2406ece0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.805+0000 7f0e46b3e640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0e401d7ee0 0x7f0e401d8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.805+0000 7f0e46b3e640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f0e40142710 0x7f0e40140480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.805+0000 7f0e46b3e640 1 --2- 172.21.15.69:0/3237503055 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e4013fa10 0x7f0e40140e90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.805+0000 7f0e46b3e640 1 -- 172.21.15.69:0/3237503055 >> 172.21.15.69:0/3237503055 conn(0x7f0e400a8a60 msgr2=0x7f0e4013f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.805+0000 7f0e46b3e640 1 -- 172.21.15.69:0/3237503055 shutdown_connections 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.805+0000 7f0e46b3e640 1 -- 172.21.15.69:0/3237503055 wait complete. 2024-01-21T04:33:29.807 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:29.805+0000 7f0e46b3e640 1 librados: shutdown 2024-01-21T04:33:29.817 DEBUG:teuthology.misc:5 of 8 OSDs are up 2024-01-21T04:33:33.958 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:33.931+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-01-21T04:33:33.980 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:33.956+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T04:33:34.077 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:33.987+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T04:33:34.097 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:34.076+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-01-21T04:33:34.173 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.039+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-21T04:33:34.173 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.087+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-01-21T04:33:34.188 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.172+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-21T04:33:34.190 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:34.096+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T04:33:34.191 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:34.128+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T04:33:34.244 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.187+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-21T04:33:34.253 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:34.189+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T04:33:34.253 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:34.200+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T04:33:34.253 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:34.211+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T04:33:34.254 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:34.241+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T04:33:34.265 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.200+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-21T04:33:34.412 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.242+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-21T04:33:34.413 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.264+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-21T04:33:34.423 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:34.325+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 3 2024-01-21T04:33:34.427 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.415+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-01-21T04:33:34.604 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.534+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-01-21T04:33:34.604 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.579+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-21T04:33:34.651 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.602+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 3 2024-01-21T04:33:34.785 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:34.423+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-01-21T04:33:34.791 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:34.499+0000 5dc2300 -1 bdev(0x18aa6200 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-01-21T04:33:34.972 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.744+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-21T04:33:34.972 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:34.846+0000 5dc2300 -1 bdev(0x18abe140 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-01-21T04:33:35.693 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:33:35.253+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-21T04:33:35.693 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:33:35.586+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-21T04:33:35.694 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:33:35.668+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-21T04:33:35.703 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:33:35.692+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-21T04:33:35.712 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:33:35.703+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 2 2024-01-21T04:33:35.778 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:33:35.712+0000 5dc2300 -1 bdev(0x18e74bf0 /var/lib/ceph/osd/ceph-3/block) aio_submit retries 1 2024-01-21T04:33:35.887 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:35.882+0000 c493640 -1 osd.6 0 waiting for initial osdmap 2024-01-21T04:33:36.006 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:36.000+0000 ac7d640 -1 osd.4 0 waiting for initial osdmap 2024-01-21T04:33:36.395 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:33:36.381+0000 1425a640 -1 osd.6 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T04:33:36.500 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:33:36.486+0000 1425a640 -1 osd.4 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T04:33:36.622 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T04:33:36.711 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.711+0000 7fe12f13a640 10 monclient: get_monmap_and_config 2024-01-21T04:33:36.712 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.711+0000 7fe12f13a640 10 monclient: build_initial_monmap 2024-01-21T04:33:36.712 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.711+0000 7fe12f13a640 10 monclient: monmap: 2024-01-21T04:33:36.712 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:36.712 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:36.712 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:33:36.712095+0000 2024-01-21T04:33:36.712 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:33:36.712095+0000 2024-01-21T04:33:36.712 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:36.712 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:36.712 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:36.712 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:36.713 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:36.713 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:36.713 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:36.713 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:36.715 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.711+0000 7fe12f13a640 1 Processor -- start 2024-01-21T04:33:36.715 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.711+0000 7fe12f13a640 1 -- start start 2024-01-21T04:33:36.715 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.711+0000 7fe12f13a640 10 monclient: init 2024-01-21T04:33:36.715 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:36.715 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-01-21T04:33:36.715 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe12813f740 0x7fe1280a9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:36.715 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 10 monclient(hunting): picked mon.noname-b con 0x7fe12813f740 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:36.715 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe1280b1b10 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:36.715 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe1280a9670 0x7fe1281429e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:36.715 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 10 monclient(hunting): picked mon.noname-c con 0x7fe1280a9670 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1280a8ee0 con 0x7fe1280b1b10 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe128142f20 con 0x7fe1280a9670 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1280aad70 con 0x7fe12813f740 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.712+0000 7fe12f13a640 10 monclient(hunting): authenticate will time out at 1375.053725s 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12d6d7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe1280a9670 0x7fe1281429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12ced6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe12813f740 0x7fe1280a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12d6d7640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe1280a9670 0x7fe1281429e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:46056/0 (socket says 172.21.15.69:46056) 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12d6d7640 1 -- 172.21.15.69:0/773443278 learned_addr learned my addr 172.21.15.69:0/773443278 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12d6d7640 10 monclient(hunting): get_auth_request con 0x7fe1280a9670 auth_method 0 2024-01-21T04:33:36.716 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12d6d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12d6d7640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12d6d7640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12ced6640 10 monclient(hunting): get_auth_request con 0x7fe12813f740 auth_method 0 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12ced6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12ced6640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.713+0000 7fe12ced6640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.714+0000 7fe12d6d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.714+0000 7fe12d6d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.714+0000 7fe12d6d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.714+0000 7fe12ced6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.714+0000 7fe12ced6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:36.717 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.714+0000 7fe12ced6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:36.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.716+0000 7fe12d6d7640 10 monclient(hunting): handle_auth_done global_id 4258 payload 274 2024-01-21T04:33:36.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.716+0000 7fe12d6d7640 1 -- 172.21.15.69:0/773443278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe12813f740 msgr2=0x7fe1280a9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:36.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.716+0000 7fe12d6d7640 1 --2- 172.21.15.69:0/773443278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe12813f740 0x7fe1280a9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:36.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.716+0000 7fe12d6d7640 1 -- 172.21.15.69:0/773443278 >> v1:172.21.15.144:6789/0 conn(0x7fe1280b1b10 legacy=0x7fe1281422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:36.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.716+0000 7fe12d6d7640 10 monclient: _finish_hunting 0 2024-01-21T04:33:36.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.716+0000 7fe12d6d7640 1 monclient: found mon.noname-c 2024-01-21T04:33:36.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.716+0000 7fe12d6d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:36.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.716+0000 7fe12d6d7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:36.718 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.716+0000 7fe12d6d7640 1 -- 172.21.15.69:0/773443278 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12813fb20 con 0x7fe1280a9670 2024-01-21T04:33:36.720 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.719+0000 7fe12d6d7640 1 --2- 172.21.15.69:0/773443278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1280a9670 0x7fe1281429e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe120009e50 tx=0x7fe12002be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51b27633f014216e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:36.724 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 1 -- 172.21.15.69:0/773443278 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12002c9e0 con 0x7fe1280a9670 2024-01-21T04:33:36.724 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:36.724 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:36.724 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 10 monclient: dump: 2024-01-21T04:33:36.724 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:36.724 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 1 -- 172.21.15.69:0/773443278 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12002cb40 con 0x7fe1280a9670 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:36.725 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe12f13a640 5 monclient: authenticate success, global_id 4258 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe12f13a640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 1 -- 172.21.15.69:0/773443278 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1200343f0 con 0x7fe1280a9670 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe12f13a640 4 set_mon_vals no callback set 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.723+0000 7fe1277fe640 10 monclient: dump: 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:36.726 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:36.727 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:36.727 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:36.730 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 10 monclient: shutdown 2024-01-21T04:33:36.730 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:36.730 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 1 -- 172.21.15.69:0/773443278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1280a9670 msgr2=0x7fe1281429e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:36.730 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 1 --2- 172.21.15.69:0/773443278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1280a9670 0x7fe1281429e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe120009e50 tx=0x7fe12002be40 comp rx=0 tx=0).stop 2024-01-21T04:33:36.730 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 1 -- 172.21.15.69:0/773443278 shutdown_connections 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 1 --2- 172.21.15.69:0/773443278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1280a9670 0x7fe1281429e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 1 --2- 172.21.15.69:0/773443278 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe12813f740 0x7fe1280a9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 1 -- 172.21.15.69:0/773443278 >> 172.21.15.69:0/773443278 conn(0x7fe128066e50 msgr2=0x7fe1280b7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 1 -- 172.21.15.69:0/773443278 shutdown_connections 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 1 -- 172.21.15.69:0/773443278 wait complete. 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 10 monclient: build_initial_monmap 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.724+0000 7fe12f13a640 10 monclient: monmap: 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:36.731 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.725+0000 7fe12f13a640 1 librados: starting msgr at 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.725+0000 7fe12f13a640 1 librados: starting objecter 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.725+0000 7fe12f13a640 1 Processor -- start 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.725+0000 7fe12f13a640 1 -- start start 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.725+0000 7fe12f13a640 1 librados: setting wanted keys 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.725+0000 7fe12f13a640 1 librados: calling monclient init 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.725+0000 7fe12f13a640 10 monclient: init 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.725+0000 7fe12f13a640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:36.732 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1280a9670 0x7fe1280b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 10 monclient(hunting): picked mon.noname-a con 0x7fe1280a9670 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1280b1b10 0x7fe1280ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 10 monclient(hunting): picked mon.noname-b con 0x7fe1280b1b10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe12813f740 0x7fe1280ab970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 10 monclient(hunting): picked mon.noname-c con 0x7fe12813f740 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12813efa0 con 0x7fe1280a9670 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1280acef0 con 0x7fe1280b1b10 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12813eac0 con 0x7fe12813f740 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:36.733 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12f13a640 10 monclient(hunting): authenticate will time out at 1375.067292s 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12ced6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1280a9670 0x7fe1280b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12d6d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe12813f740 0x7fe1280ab970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe127fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1280b1b10 0x7fe1280ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12ced6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1280a9670 0x7fe1280b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47020/0 (socket says 172.21.15.69:47020) 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12ced6640 1 -- 172.21.15.69:0/4211460498 learned_addr learned my addr 172.21.15.69:0/4211460498 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12ced6640 10 monclient(hunting): get_auth_request con 0x7fe1280a9670 auth_method 0 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12ced6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12ced6640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.726+0000 7fe12ced6640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe127fff640 10 monclient(hunting): get_auth_request con 0x7fe1280b1b10 auth_method 0 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe127fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe127fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe127fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:36.734 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe12d6d7640 10 monclient(hunting): get_auth_request con 0x7fe12813f740 auth_method 0 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe12d6d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe12d6d7640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe12d6d7640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe12ced6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe12ced6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe12ced6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe127fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe127fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe127fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe12d6d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe12d6d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:36.735 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.727+0000 7fe12d6d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:36.737 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.730+0000 7fe12ced6640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-01-21T04:33:36.737 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.730+0000 7fe12ced6640 1 -- 172.21.15.69:0/4211460498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe12813f740 msgr2=0x7fe1280ab970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:36.737 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.730+0000 7fe12ced6640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe12813f740 0x7fe1280ab970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:36.737 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.730+0000 7fe12ced6640 1 -- 172.21.15.69:0/4211460498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1280b1b10 msgr2=0x7fe1280ab430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:36.737 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.730+0000 7fe12ced6640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1280b1b10 0x7fe1280ab430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:36.737 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.730+0000 7fe12ced6640 10 monclient: _finish_hunting 0 2024-01-21T04:33:36.737 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.730+0000 7fe12ced6640 1 monclient: found mon.noname-a 2024-01-21T04:33:36.737 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.730+0000 7fe12ced6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:36.737 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.730+0000 7fe12ced6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.730+0000 7fe12ced6640 1 -- 172.21.15.69:0/4211460498 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe120009b30 con 0x7fe1280a9670 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.733+0000 7fe12ced6640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1280a9670 0x7fe1280b00d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe11c00e9b0 tx=0x7fe11c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.733+0000 7fe1257fa640 10 client.?.objecter ms_handle_connect 0x7fe1280a9670 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.733+0000 7fe1257fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe1257fa640 1 -- 172.21.15.69:0/4211460498 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c009670 con 0x7fe1280a9670 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe1257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe1257fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe1257fa640 10 monclient: dump: 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:36.738 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe1257fa640 10 monclient: _finish_auth 0 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe1257fa640 10 monclient: _check_auth_tickets 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe1257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe12f13a640 5 monclient: authenticate success, global_id 4251 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe12f13a640 10 monclient: _renew_subs 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe12f13a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe12f13a640 1 -- 172.21.15.69:0/4211460498 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12813f3d0 con 0x7fe1280a9670 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe12f13a640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe12f13a640 10 monclient: _renew_subs 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe12f13a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:36.739 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe12f13a640 1 -- 172.21.15.69:0/4211460498 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1281dae70 con 0x7fe1280a9670 2024-01-21T04:33:36.740 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.736+0000 7fe12f13a640 1 librados: init done 2024-01-21T04:33:36.740 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.737+0000 7fe12f13a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:36.740 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.737+0000 7fe12f13a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:36.740 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.737+0000 7fe12f13a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:36.741 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.737+0000 7fe12f13a640 1 -- 172.21.15.69:0/4211460498 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12813f210 con 0x7fe1280a9670 2024-01-21T04:33:36.741 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.738+0000 7fe1257fa640 1 -- 172.21.15.69:0/4211460498 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11c00c9a0 con 0x7fe1280a9670 2024-01-21T04:33:36.741 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.738+0000 7fe1257fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:36.741 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.738+0000 7fe126ffd640 4 set_mon_vals no callback set 2024-01-21T04:33:36.996 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.995+0000 7fe1257fa640 1 -- 172.21.15.69:0/4211460498 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c00cd10 con 0x7fe1280a9670 2024-01-21T04:33:36.996 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.995+0000 7fe1257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:36.996 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.995+0000 7fe1257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:36.996 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.995+0000 7fe1257fa640 10 monclient: dump: 2024-01-21T04:33:36.996 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:36.996 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:36.996 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:36.996 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:36.996 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:36.996 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:36.997 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:36.997 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:36.997 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:36.997 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:36.998 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.996+0000 7fe1257fa640 1 -- 172.21.15.69:0/4211460498 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) v1 ==== 89142+0+0 (secure 0 0 0) 0x7fe11c013470 con 0x7fe1280a9670 2024-01-21T04:33:36.998 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.996+0000 7fe1257fa640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10c06c7d0 0x7fe10c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:36.998 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.997+0000 7fe127fff640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10c06c7d0 0x7fe10c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:36.998 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.997+0000 7fe127fff640 10 monclient: get_auth_request con 0x7fe10c06c7d0 auth_method 0 2024-01-21T04:33:36.998 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.997+0000 7fe127fff640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10c06c7d0 0x7fe10c06ec90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe1280aca60 tx=0x7fe114007e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:36.998 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:36.997+0000 7fe1257fa640 10 client.4251.objecter ms_handle_connect 0x7fe10c06c7d0 2024-01-21T04:33:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.013+0000 7fe1257fa640 1 -- 172.21.15.69:0/4211460498 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4017+0+0 (secure 0 0 0) 0x7fe11c058980 con 0x7fe1280a9670 2024-01-21T04:33:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.013+0000 7fe1257fa640 10 client.4251.objecter ms_dispatch 0x7fe128000b90 osd_map(19..19 src has 1..19) v4 2024-01-21T04:33:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.013+0000 7fe1257fa640 3 client.4251.objecter handle_osd_map got epochs [19,19] > 0 2024-01-21T04:33:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.013+0000 7fe1257fa640 3 client.4251.objecter handle_osd_map decoding full epoch 19 2024-01-21T04:33:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.013+0000 7fe1257fa640 20 client.4251.objecter dump_active .. 0 homeless 2024-01-21T04:33:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.013+0000 7fe1257fa640 1 -- 172.21.15.69:0/4211460498 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe11c054cc0 con 0x7fe1280a9670 2024-01-21T04:33:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.013+0000 7fe1257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:37.014 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.013+0000 7fe1257fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:37.020 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:33:37.016+0000 ac7d640 -1 osd.3 0 waiting for initial osdmap 2024-01-21T04:33:37.212 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.211+0000 7fe12f13a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:37.213 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.211+0000 7fe12f13a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:37.213 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.211+0000 7fe12f13a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:37.213 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.211+0000 7fe12f13a640 1 -- 172.21.15.69:0/4211460498 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe12813f210 con 0x7fe1280a9670 2024-01-21T04:33:37.393 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.391+0000 7fe1257fa640 1 -- 172.21.15.69:0/4211460498 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+11312 (secure 0 0 0) 0x7fe12813f210 con 0x7fe1280a9670 2024-01-21T04:33:37.393 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.392+0000 7fe1257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:37.393 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.392+0000 7fe1257fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:37.393 INFO:teuthology.misc.health.smithi069.stdout: 2024-01-21T04:33:37.393 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":19,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T04:33:33.722688+0000","last_up_change":"2024-01-21T04:33:28.929791+0000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-21T04:33:29.939892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6825","nonce":335500148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6827","nonce":335500148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6831","nonce":335500148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6829","nonce":335500148}]},"public_addr":"172.21.15.69:6825/335500148","cluster_addr":"172.21.15.69:6827/335500148","heartbeat_back_addr":"172.21.15.69:6831/335500148","heartbeat_front_addr":"172.21.15.69:6829/335500148","state":["exists","up"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6809","nonce":3868480563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6811","nonce":3868480563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6815","nonce":3868480563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6813","nonce":3868480563}]},"public_addr":"172.21.15.69:6809/3868480563","cluster_addr":"172.21.15.69:6811/3868480563","heartbeat_back_addr":"172.21.15.69:6815/3868480563","heartbeat_front_addr":"172.21.15.69:6813/3868480563","state":["exists","up"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6817","nonce":3954012353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6819","nonce":3954012353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6823","nonce":3954012353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6821","nonce":3954012353}]},"public_addr":"172.21.15.69:6817/3954012353","cluster_addr":"172.21.15.69:6819/3954012353","heartbeat_back_addr":"172.21.15.69:6823/3954012353","heartbeat_front_addr":"172.21.15.69:6821/3954012353","state":["exists","up"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6803","nonce":1782989653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6805","nonce":1782989653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6809","nonce":1782989653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6807","nonce":1782989653}]},"public_addr":"172.21.15.144:6803/1782989653","cluster_addr":"172.21.15.144:6805/1782989653","heartbeat_back_addr":"172.21.15.144:6809/1782989653","heartbeat_front_addr":"172.21.15.144:6807/1782989653","state":["exists","up"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6811","nonce":14156089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6815","nonce":14156089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6823","nonce":14156089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6819","nonce":14156089}]},"public_addr":"172.21.15.144:6811/14156089","cluster_addr":"172.21.15.144:6815/14156089","heartbeat_back_addr":"172.21.15.144:6823/14156089","heartbeat_front_addr":"172.21.15.144:6819/14156089","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:27.418834+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.672251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.150659+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:24.415555+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.940928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T04:33:37.395 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 10 librados: watch_flush enter 2024-01-21T04:33:37.395 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 10 librados: watch_flush exit 2024-01-21T04:33:37.395 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 20 client.4251.objecter shutdown clearing up homeless session... 2024-01-21T04:33:37.395 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 10 client.4251.objecter successfully canceled tick 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 -- 172.21.15.69:0/4211460498 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10c06c7d0 msgr2=0x7fe10c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10c06c7d0 0x7fe10c06ec90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe1280aca60 tx=0x7fe114007e30 comp rx=0 tx=0).stop 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 10 monclient: shutdown 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 -- 172.21.15.69:0/4211460498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1280a9670 msgr2=0x7fe1280b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1280a9670 0x7fe1280b00d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe11c00e9b0 tx=0x7fe11c00ee70 comp rx=0 tx=0).stop 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 -- 172.21.15.69:0/4211460498 shutdown_connections 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10c06c7d0 0x7fe10c06ec90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe12813f740 0x7fe1280ab970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1280b1b10 0x7fe1280ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 --2- 172.21.15.69:0/4211460498 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1280a9670 0x7fe1280b00d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 -- 172.21.15.69:0/4211460498 >> 172.21.15.69:0/4211460498 conn(0x7fe128066e50 msgr2=0x7fe1280b6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 -- 172.21.15.69:0/4211460498 shutdown_connections 2024-01-21T04:33:37.396 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 -- 172.21.15.69:0/4211460498 wait complete. 2024-01-21T04:33:37.397 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:37.394+0000 7fe12f13a640 1 librados: shutdown 2024-01-21T04:33:37.407 DEBUG:teuthology.misc:5 of 8 OSDs are up 2024-01-21T04:33:37.507 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:33:37.491+0000 1425a640 -1 osd.3 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-21T04:33:41.285 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:09.192 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:33:41.346 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:01:09.221 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:33:41.369 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:09.276 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:33:41.377 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:09.284 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:33:41.402 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:01:09.277 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:33:41.442 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:09.349 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:33:41.633 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:01:09.508 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:33:41.644 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:09.551 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:33:41.700 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:09.607 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:33:44.213 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-21T04:33:44.301 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.300+0000 7f630ede2640 10 monclient: get_monmap_and_config 2024-01-21T04:33:44.301 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.300+0000 7f630ede2640 10 monclient: build_initial_monmap 2024-01-21T04:33:44.301 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.300+0000 7f630ede2640 10 monclient: monmap: 2024-01-21T04:33:44.301 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:44.301 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:44.301 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:33:44.301187+0000 2024-01-21T04:33:44.301 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:33:44.301187+0000 2024-01-21T04:33:44.301 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:44.301 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.302 INFO:teuthology.misc.health.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:44.302 INFO:teuthology.misc.health.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:44.302 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:44.302 INFO:teuthology.misc.health.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:44.302 INFO:teuthology.misc.health.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:44.302 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:44.304 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.300+0000 7f630ede2640 1 Processor -- start 2024-01-21T04:33:44.304 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.300+0000 7f630ede2640 1 -- start start 2024-01-21T04:33:44.304 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.300+0000 7f630ede2640 10 monclient: init 2024-01-21T04:33:44.304 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:44.304 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-01-21T04:33:44.304 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f63080a8a60 0x7f630813f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:44.304 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 10 monclient(hunting): picked mon.noname-c con 0x7f63080a8a60 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f630813f840 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f63081428e0 0x7f630814c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 10 monclient(hunting): picked mon.noname-a con 0x7f63081428e0 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63080b1de0 con 0x7f630813f840 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63080a8e40 con 0x7f63081428e0 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630813fc70 con 0x7f63080a8a60 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.301+0000 7f630ede2640 10 monclient(hunting): authenticate will time out at 1382.642753s 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.302+0000 7f630d37f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f63081428e0 0x7f630814c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.302+0000 7f630cb7e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f63080a8a60 0x7f630813f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.302+0000 7f630d37f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f63081428e0 0x7f630814c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47026/0 (socket says 172.21.15.69:47026) 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.302+0000 7f630d37f640 1 -- 172.21.15.69:0/805197600 learned_addr learned my addr 172.21.15.69:0/805197600 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:44.305 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.302+0000 7f630d37f640 10 monclient(hunting): get_auth_request con 0x7f63081428e0 auth_method 0 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.302+0000 7f630d37f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.302+0000 7f630d37f640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.302+0000 7f630d37f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.303+0000 7f630cb7e640 10 monclient(hunting): get_auth_request con 0x7f63080a8a60 auth_method 0 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.303+0000 7f630cb7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.303+0000 7f630cb7e640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.303+0000 7f630cb7e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.303+0000 7f630d37f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.303+0000 7f630d37f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.303+0000 7f630d37f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.303+0000 7f630cb7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.303+0000 7f630cb7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:44.306 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.303+0000 7f630cb7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:44.307 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.304+0000 7f630d37f640 10 monclient(hunting): handle_auth_done global_id 4257 payload 274 2024-01-21T04:33:44.307 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.304+0000 7f630d37f640 1 -- 172.21.15.69:0/805197600 >> v2:172.21.15.144:3300/0 conn(0x7f63080a8a60 msgr2=0x7f630813f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:44.307 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.304+0000 7f630d37f640 1 --2- 172.21.15.69:0/805197600 >> v2:172.21.15.144:3300/0 conn(0x7f63080a8a60 0x7f630813f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.307 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.304+0000 7f630d37f640 1 -- 172.21.15.69:0/805197600 >> v1:172.21.15.144:6789/0 conn(0x7f630813f840 legacy=0x7f63081421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:44.307 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.304+0000 7f630d37f640 10 monclient: _finish_hunting 0 2024-01-21T04:33:44.307 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.304+0000 7f630d37f640 1 monclient: found mon.noname-a 2024-01-21T04:33:44.307 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.304+0000 7f630d37f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:44.308 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.304+0000 7f630d37f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:44.308 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.304+0000 7f630d37f640 1 -- 172.21.15.69:0/805197600 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080aa480 con 0x7f63081428e0 2024-01-21T04:33:44.308 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.306+0000 7f630d37f640 1 --2- 172.21.15.69:0/805197600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63081428e0 0x7f630814c930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f63080b53b0 tx=0x7f630002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2729eaecc5559955 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:44.312 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 1 -- 172.21.15.69:0/805197600 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630002ca80 con 0x7f63081428e0 2024-01-21T04:33:44.312 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:44.312 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:44.312 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 10 monclient: dump: 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:44.313 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 1 -- 172.21.15.69:0/805197600 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630002cbe0 con 0x7f63081428e0 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f630ede2640 5 monclient: authenticate success, global_id 4257 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f630ede2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f630ede2640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 1 -- 172.21.15.69:0/805197600 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63000343f0 con 0x7f63081428e0 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f630ede2640 4 set_mon_vals no callback set 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f63077fe640 10 monclient: dump: 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:44.314 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f630ede2640 10 monclient: shutdown 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f630ede2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f630ede2640 1 -- 172.21.15.69:0/805197600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63081428e0 msgr2=0x7f630814c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.310+0000 7f630ede2640 1 --2- 172.21.15.69:0/805197600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63081428e0 0x7f630814c930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f63080b53b0 tx=0x7f630002bee0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.311+0000 7f630ede2640 1 -- 172.21.15.69:0/805197600 shutdown_connections 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.311+0000 7f630ede2640 1 --2- 172.21.15.69:0/805197600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63081428e0 0x7f630814c930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.311+0000 7f630ede2640 1 --2- 172.21.15.69:0/805197600 >> v2:172.21.15.144:3300/0 conn(0x7f63080a8a60 0x7f630813f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.311+0000 7f630ede2640 1 -- 172.21.15.69:0/805197600 >> 172.21.15.69:0/805197600 conn(0x7f63080670b0 msgr2=0x7f63080b78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.311+0000 7f630ede2640 1 -- 172.21.15.69:0/805197600 shutdown_connections 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.311+0000 7f630ede2640 1 -- 172.21.15.69:0/805197600 wait complete. 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.311+0000 7f630ede2640 10 monclient: build_initial_monmap 2024-01-21T04:33:44.315 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.311+0000 7f630ede2640 10 monclient: monmap: 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:epoch 0 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:created 0.000000 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.311+0000 7f630ede2640 1 librados: starting msgr at 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.311+0000 7f630ede2640 1 librados: starting objecter 2024-01-21T04:33:44.316 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 1 Processor -- start 2024-01-21T04:33:44.317 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 1 -- start start 2024-01-21T04:33:44.317 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 1 librados: setting wanted keys 2024-01-21T04:33:44.317 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 1 librados: calling monclient init 2024-01-21T04:33:44.317 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 10 monclient: init 2024-01-21T04:33:44.321 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f630813f840 0x7f6308140b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 10 monclient(hunting): picked mon.noname-b con 0x7f630813f840 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63081428e0 0x7f6308141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 10 monclient(hunting): picked mon.noname-a con 0x7f63081428e0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63081446c0 0x7f6308144b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 10 monclient(hunting): picked mon.noname-c con 0x7f63081446c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6308143a70 con 0x7f63081428e0 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6308141e70 con 0x7f630813f840 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6308143ce0 con 0x7f63081446c0 2024-01-21T04:33:44.322 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.312+0000 7f630ede2640 10 monclient(hunting): authenticate will time out at 1382.654087s 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f6307fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63081428e0 0x7f6308141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f630cb7e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f630813f840 0x7f6308140b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f630d37f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63081446c0 0x7f6308144b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f6307fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63081428e0 0x7f6308141330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47028/0 (socket says 172.21.15.69:47028) 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f6307fff640 1 -- 172.21.15.69:0/697088559 learned_addr learned my addr 172.21.15.69:0/697088559 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f6307fff640 10 monclient(hunting): get_auth_request con 0x7f63081428e0 auth_method 0 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f6307fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f6307fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f6307fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f630cb7e640 10 monclient(hunting): get_auth_request con 0x7f630813f840 auth_method 0 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f630cb7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f630cb7e640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:44.323 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f630cb7e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f630d37f640 10 monclient(hunting): get_auth_request con 0x7f63081446c0 auth_method 0 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f630d37f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f630d37f640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.313+0000 7f630d37f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.314+0000 7f6307fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.314+0000 7f6307fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.314+0000 7f6307fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.314+0000 7f630cb7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.314+0000 7f630cb7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.314+0000 7f630cb7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.314+0000 7f630d37f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.314+0000 7f630d37f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.314+0000 7f630d37f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:44.324 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.316+0000 7f630cb7e640 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.316+0000 7f630cb7e640 1 -- 172.21.15.69:0/697088559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63081446c0 msgr2=0x7f6308144b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.316+0000 7f630cb7e640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63081446c0 0x7f6308144b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.316+0000 7f630cb7e640 1 -- 172.21.15.69:0/697088559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63081428e0 msgr2=0x7f6308141330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.316+0000 7f630cb7e640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63081428e0 0x7f6308141330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.316+0000 7f630cb7e640 10 monclient: _finish_hunting 0 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.316+0000 7f630cb7e640 1 monclient: found mon.noname-b 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.316+0000 7f630cb7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.316+0000 7f630cb7e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.316+0000 7f630cb7e640 1 -- 172.21.15.69:0/697088559 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6300009b30 con 0x7f630813f840 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.319+0000 7f630cb7e640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f630813f840 0x7f6308140b90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00e850 tx=0x7f62fc00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.319+0000 7f63057fa640 10 client.?.objecter ms_handle_connect 0x7f630813f840 2024-01-21T04:33:44.325 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.319+0000 7f63057fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:44.326 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.322+0000 7f63057fa640 1 -- 172.21.15.69:0/697088559 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0095b0 con 0x7f630813f840 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 10 monclient: dump: 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:44.327 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 10 monclient: _finish_auth 0 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 10 monclient: _check_auth_tickets 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f630ede2640 5 monclient: authenticate success, global_id 4273 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 1 -- 172.21.15.69:0/697088559 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc00c940 con 0x7f630813f840 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f630ede2640 10 monclient: _renew_subs 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 1 -- 172.21.15.69:0/697088559 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc00ccd0 con 0x7f630813f840 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f630ede2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f630ede2640 1 -- 172.21.15.69:0/697088559 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6308143fc0 con 0x7f630813f840 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:44.328 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f6306ffd640 4 set_mon_vals no callback set 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f63057fa640 10 monclient: dump: 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:epoch 1 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr: 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f630ede2640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:44.329 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f630ede2640 10 monclient: _renew_subs 2024-01-21T04:33:44.330 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f630ede2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:44.330 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f630ede2640 1 -- 172.21.15.69:0/697088559 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6308145090 con 0x7f630813f840 2024-01-21T04:33:44.330 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.323+0000 7f630ede2640 1 librados: init done 2024-01-21T04:33:44.330 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.324+0000 7f630ede2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:44.330 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.324+0000 7f630ede2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:44.330 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.324+0000 7f630ede2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:44.330 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.324+0000 7f630ede2640 1 -- 172.21.15.69:0/697088559 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63080a9cf0 con 0x7f630813f840 2024-01-21T04:33:44.566 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.565+0000 7f63057fa640 1 -- 172.21.15.69:0/697088559 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f62fc013470 con 0x7f630813f840 2024-01-21T04:33:44.566 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.566+0000 7f63057fa640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f62ec06c8b0 0x7f62ec06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:44.567 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.566+0000 7f6307fff640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f62ec06c8b0 0x7f62ec06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:44.567 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.566+0000 7f6307fff640 10 monclient: get_auth_request con 0x7f62ec06c8b0 auth_method 0 2024-01-21T04:33:44.567 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.566+0000 7f63057fa640 10 client.4273.objecter ms_handle_connect 0x7f62ec06c8b0 2024-01-21T04:33:44.567 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.566+0000 7f6307fff640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f62ec06c8b0 0x7f62ec06ed70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f63080a98c0 tx=0x7f62f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:44.569 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.568+0000 7f63057fa640 1 -- 172.21.15.69:0/697088559 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f62fc00ce30 con 0x7f630813f840 2024-01-21T04:33:44.569 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.568+0000 7f63057fa640 10 client.4273.objecter ms_dispatch 0x7f6308000b90 osd_map(22..22 src has 1..22) v4 2024-01-21T04:33:44.569 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.568+0000 7f63057fa640 3 client.4273.objecter handle_osd_map got epochs [22,22] > 0 2024-01-21T04:33:44.569 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.568+0000 7f63057fa640 3 client.4273.objecter handle_osd_map decoding full epoch 22 2024-01-21T04:33:44.569 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.568+0000 7f63057fa640 20 client.4273.objecter dump_active .. 0 homeless 2024-01-21T04:33:44.578 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.578+0000 7f63057fa640 1 -- 172.21.15.69:0/697088559 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f62fc054e70 con 0x7f630813f840 2024-01-21T04:33:44.578 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.578+0000 7f63057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:44.579 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.578+0000 7f63057fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:44.749 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.748+0000 7f630ede2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:44.749 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.748+0000 7f630ede2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:44.749 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.748+0000 7f630ede2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:44.749 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.748+0000 7f630ede2640 1 -- 172.21.15.69:0/697088559 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6308075c30 con 0x7f630813f840 2024-01-21T04:33:44.835 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.834+0000 7f63057fa640 1 -- 172.21.15.69:0/697088559 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+12815 (secure 0 0 0) 0x7f62fc021090 con 0x7f630813f840 2024-01-21T04:33:44.835 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.834+0000 7f63057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:44.835 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.834+0000 7f63057fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:44.836 INFO:teuthology.misc.health.smithi069.stdout: 2024-01-21T04:33:44.836 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":22,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T04:33:39.982082+0000","last_up_change":"2024-01-21T04:33:38.448164+0000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-21T04:33:29.939892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6825","nonce":335500148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6827","nonce":335500148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6831","nonce":335500148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6829","nonce":335500148}]},"public_addr":"172.21.15.69:6825/335500148","cluster_addr":"172.21.15.69:6827/335500148","heartbeat_back_addr":"172.21.15.69:6831/335500148","heartbeat_front_addr":"172.21.15.69:6829/335500148","state":["exists","up"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6809","nonce":3868480563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6811","nonce":3868480563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6815","nonce":3868480563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6813","nonce":3868480563}]},"public_addr":"172.21.15.69:6809/3868480563","cluster_addr":"172.21.15.69:6811/3868480563","heartbeat_back_addr":"172.21.15.69:6815/3868480563","heartbeat_front_addr":"172.21.15.69:6813/3868480563","state":["exists","up"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6817","nonce":3954012353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6819","nonce":3954012353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6823","nonce":3954012353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6821","nonce":3954012353}]},"public_addr":"172.21.15.69:6817/3954012353","cluster_addr":"172.21.15.69:6819/3954012353","heartbeat_back_addr":"172.21.15.69:6823/3954012353","heartbeat_front_addr":"172.21.15.69:6821/3954012353","state":["exists","up"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6801","nonce":1086763384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6803","nonce":1086763384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6807","nonce":1086763384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6805","nonce":1086763384}]},"public_addr":"172.21.15.69:6801/1086763384","cluster_addr":"172.21.15.69:6803/1086763384","heartbeat_back_addr":"172.21.15.69:6807/1086763384","heartbeat_front_addr":"172.21.15.69:6805/1086763384","state":["exists","up"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6827","nonce":3897583001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6829","nonce":3897583001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6833","nonce":3897583001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6831","nonce":3897583001}]},"public_addr":"172.21.15.144:6827/3897583001","cluster_addr":"172.21.15.144:6829/3897583001","heartbeat_back_addr":"172.21.15.144:6833/3897583001","heartbeat_front_addr":"172.21.15.144:6831/3897583001","state":["exists","up"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6803","nonce":1782989653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6805","nonce":1782989653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6809","nonce":1782989653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6807","nonce":1782989653}]},"public_addr":"172.21.15.144:6803/1782989653","cluster_addr":"172.21.15.144:6805/1782989653","heartbeat_back_addr":"172.21.15.144:6809/1782989653","heartbeat_front_addr":"172.21.15.144:6807/1782989653","state":["exists","up"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6813","nonce":479183693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6818","nonce":479183693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6825","nonce":479183693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6822","nonce":479183693}]},"public_addr":"172.21.15.144:6813/479183693","cluster_addr":"172.21.15.144:6818/479183693","heartbeat_back_addr":"172.21.15.144:6825/479183693","heartbeat_front_addr":"172.21.15.144:6822/479183693","state":["exists","up"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6811","nonce":14156089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6815","nonce":14156089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6823","nonce":14156089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6819","nonce":14156089}]},"public_addr":"172.21.15.144:6811/14156089","cluster_addr":"172.21.15.144:6815/14156089","heartbeat_back_addr":"172.21.15.144:6823/14156089","heartbeat_front_addr":"172.21.15.144:6819/14156089","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:27.418834+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.672251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.150659+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.544731+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.167108+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:24.415555+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.272052+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.940928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T04:33:44.838 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 10 librados: watch_flush enter 2024-01-21T04:33:44.838 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 10 librados: watch_flush exit 2024-01-21T04:33:44.838 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 20 client.4273.objecter shutdown clearing up homeless session... 2024-01-21T04:33:44.838 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 10 client.4273.objecter successfully canceled tick 2024-01-21T04:33:44.838 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 -- 172.21.15.69:0/697088559 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f62ec06c8b0 msgr2=0x7f62ec06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f62ec06c8b0 0x7f62ec06ed70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f63080a98c0 tx=0x7f62f4009040 comp rx=0 tx=0).stop 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 10 monclient: shutdown 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 -- 172.21.15.69:0/697088559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f630813f840 msgr2=0x7f6308140b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f630813f840 0x7f6308140b90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00e850 tx=0x7f62fc00ed10 comp rx=0 tx=0).stop 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 -- 172.21.15.69:0/697088559 shutdown_connections 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f62ec06c8b0 0x7f62ec06ed70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63081446c0 0x7f6308144b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63081428e0 0x7f6308141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 --2- 172.21.15.69:0/697088559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f630813f840 0x7f6308140b90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 -- 172.21.15.69:0/697088559 >> 172.21.15.69:0/697088559 conn(0x7f63080a8a60 msgr2=0x7f6308073df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 -- 172.21.15.69:0/697088559 shutdown_connections 2024-01-21T04:33:44.839 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 -- 172.21.15.69:0/697088559 wait complete. 2024-01-21T04:33:44.840 INFO:teuthology.misc.health.smithi069.stderr:2024-01-21T04:33:44.836+0000 7f630ede2640 1 librados: shutdown 2024-01-21T04:33:44.848 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-21T04:33:44.848 INFO:tasks.ceph:Creating RBD pool 2024-01-21T04:33:44.848 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-21T04:33:44.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.947+0000 7f4035a07640 10 monclient: get_monmap_and_config 2024-01-21T04:33:44.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.947+0000 7f4035a07640 10 monclient: build_initial_monmap 2024-01-21T04:33:44.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.947+0000 7f4035a07640 10 monclient: monmap: 2024-01-21T04:33:44.948 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:44.948106+0000 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:44.948106+0000 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.947+0000 7f4035a07640 1 Processor -- start 2024-01-21T04:33:44.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.947+0000 7f4035a07640 1 -- start start 2024-01-21T04:33:44.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.947+0000 7f4035a07640 10 monclient: init 2024-01-21T04:33:44.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f40300a8b20 0x7f4030145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 10 monclient(hunting): picked mon.noname-a con 0x7f40300a8b20 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40300a9d10 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40300b2310 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40301456f0 con 0x7f40300a9d10 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40300a98c0 con 0x7f40300b2310 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40301460c0 con 0x7f40300a8b20 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f4035a07640 10 monclient(hunting): authenticate will time out at 1383.289512s 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f402f7fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f40300b2310 0x7f403014c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:53904/0 (socket says 172.21.15.69:53904) 2024-01-21T04:33:44.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.948+0000 7f402f7fe640 1 -- 172.21.15.69:0/4054378894 learned_addr learned my addr 172.21.15.69:0/4054378894 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.949+0000 7f402effd640 1 --2- 172.21.15.69:0/4054378894 >> v2:172.21.15.69:3300/0 conn(0x7f40300a8b20 0x7f4030145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.949+0000 7f402effd640 10 monclient(hunting): get_auth_request con 0x7f40300a8b20 auth_method 0 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.949+0000 7f402effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.949+0000 7f402effd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.949+0000 7f402effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.950+0000 7f402effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.950+0000 7f402effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.950+0000 7f402effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.951+0000 7f402dffb640 1 -- 172.21.15.69:0/4054378894 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148499360 0 0) 0x7f40300a98c0 con 0x7f40300b2310 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.951+0000 7f402dffb640 10 monclient(hunting): _negotiate 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.951+0000 7f402dffb640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.951+0000 7f402dffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.951+0000 7f402dffb640 10 monclient(hunting): my global_id is 4276 2024-01-21T04:33:44.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.951+0000 7f402dffb640 1 -- 172.21.15.69:0/4054378894 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4018003520 con 0x7f40300b2310 2024-01-21T04:33:44.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.952+0000 7f402dffb640 1 -- 172.21.15.69:0/4054378894 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271069824 0 0) 0x7f40301456f0 con 0x7f40300a9d10 2024-01-21T04:33:44.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.952+0000 7f402dffb640 10 monclient(hunting): _negotiate 2024-01-21T04:33:44.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.952+0000 7f402dffb640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:44.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.952+0000 7f402dffb640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:44.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.952+0000 7f402dffb640 10 monclient(hunting): my global_id is 4266 2024-01-21T04:33:44.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.952+0000 7f402dffb640 1 -- 172.21.15.69:0/4054378894 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40300a98c0 con 0x7f40300a9d10 2024-01-21T04:33:44.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.954+0000 7f402effd640 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2024-01-21T04:33:44.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.954+0000 7f402effd640 1 -- 172.21.15.69:0/4054378894 >> v1:172.21.15.144:6789/0 conn(0x7f40300b2310 legacy=0x7f403014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:44.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.954+0000 7f402dffb640 1 -- 172.21.15.69:0/4054378894 <== mon.1 v1:172.21.15.144:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 52793641 0 0) 0x7f4018003520 con 0x7f40300b2310 2024-01-21T04:33:44.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.954+0000 7f402effd640 1 -- 172.21.15.69:0/4054378894 >> v1:172.21.15.69:6789/0 conn(0x7f40300a9d10 legacy=0x7f403013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:44.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.954+0000 7f402effd640 10 monclient: _finish_hunting 0 2024-01-21T04:33:44.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.954+0000 7f402effd640 1 monclient: found mon.noname-a 2024-01-21T04:33:44.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.954+0000 7f402effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:44.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.954+0000 7f402effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:44.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.954+0000 7f402effd640 1 -- 172.21.15.69:0/4054378894 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4030146200 con 0x7f40300a8b20 2024-01-21T04:33:44.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.954+0000 7f402dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-21T04:33:44.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.959+0000 7f402effd640 1 --2- 172.21.15.69:0/4054378894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40300a8b20 0x7f4030145b80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4020009720 tx=0x7f402002b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbae2816dedf2ff2 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:44.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 1 -- 172.21.15.69:0/4054378894 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40280019f0 con 0x7f40300a8b20 2024-01-21T04:33:44.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:44.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:44.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 10 monclient: dump: 2024-01-21T04:33:44.964 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:44.964 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:44.964 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 10 monclient: _finish_auth 0 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 10 monclient: _check_auth_tickets 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 1 -- 172.21.15.69:0/4054378894 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402002e050 con 0x7f40300a8b20 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f4035a07640 5 monclient: authenticate success, global_id 4263 2024-01-21T04:33:44.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 1 -- 172.21.15.69:0/4054378894 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020030730 con 0x7f40300a8b20 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f402dffb640 10 monclient: dump: 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f4035a07640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.963+0000 7f4035a07640 4 set_mon_vals no callback set 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.964+0000 7f4035a07640 10 monclient: shutdown 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.964+0000 7f4035a07640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.964+0000 7f4035a07640 1 -- 172.21.15.69:0/4054378894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40300a8b20 msgr2=0x7f4030145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.964+0000 7f4035a07640 1 --2- 172.21.15.69:0/4054378894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40300a8b20 0x7f4030145b80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4020009720 tx=0x7f402002b730 comp rx=0 tx=0).stop 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.964+0000 7f4035a07640 1 -- 172.21.15.69:0/4054378894 shutdown_connections 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.964+0000 7f4035a07640 1 --2- 172.21.15.69:0/4054378894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40300a8b20 0x7f4030145b80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.964+0000 7f4035a07640 1 -- 172.21.15.69:0/4054378894 >> 172.21.15.69:0/4054378894 conn(0x7f4030066fb0 msgr2=0x7f40300b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.964+0000 7f4035a07640 1 -- 172.21.15.69:0/4054378894 shutdown_connections 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.964+0000 7f4035a07640 1 -- 172.21.15.69:0/4054378894 wait complete. 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.964+0000 7f4035a07640 10 monclient: build_initial_monmap 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.964+0000 7f4035a07640 10 monclient: monmap: 2024-01-21T04:33:44.971 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.965+0000 7f4035a07640 1 librados: starting msgr at 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.965+0000 7f4035a07640 1 librados: starting objecter 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.965+0000 7f4035a07640 1 Processor -- start 2024-01-21T04:33:44.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.965+0000 7f4035a07640 1 -- start start 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.965+0000 7f4035a07640 1 librados: setting wanted keys 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.965+0000 7f4035a07640 1 librados: calling monclient init 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.965+0000 7f4035a07640 10 monclient: init 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40300a8b20 0x7f40301448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 10 monclient(hunting): picked mon.noname-b con 0x7f40300a8b20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40300a9d10 0x7f4030135ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 10 monclient(hunting): picked mon.noname-c con 0x7f40300a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40300b2310 0x7f4030140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 10 monclient(hunting): picked mon.noname-a con 0x7f40300b2310 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4030140580 con 0x7f40300b2310 2024-01-21T04:33:44.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4030141610 con 0x7f40300a8b20 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4030136110 con 0x7f40300a9d10 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f4035a07640 10 monclient(hunting): authenticate will time out at 1383.307367s 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40300b2310 0x7f4030140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f402e7fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40300a9d10 0x7f4030135ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f402effd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40300a8b20 0x7f40301448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f402f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40300b2310 0x7f4030140040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53124/0 (socket says 172.21.15.69:53124) 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f402f7fe640 1 -- 172.21.15.69:0/3128389520 learned_addr learned my addr 172.21.15.69:0/3128389520 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f402f7fe640 10 monclient(hunting): get_auth_request con 0x7f40300b2310 auth_method 0 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f402f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f402f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:44.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.966+0000 7f402f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402effd640 10 monclient(hunting): get_auth_request con 0x7f40300a8b20 auth_method 0 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402effd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402e7fc640 10 monclient(hunting): get_auth_request con 0x7f40300a9d10 auth_method 0 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402e7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:44.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.967+0000 7f402e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.969+0000 7f402effd640 10 monclient(hunting): handle_auth_done global_id 4279 payload 824 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.969+0000 7f402effd640 1 -- 172.21.15.69:0/3128389520 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40300a9d10 msgr2=0x7f4030135ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.969+0000 7f402effd640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40300a9d10 0x7f4030135ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.969+0000 7f402effd640 1 -- 172.21.15.69:0/3128389520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40300b2310 msgr2=0x7f4030140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.969+0000 7f402effd640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40300b2310 0x7f4030140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.969+0000 7f402effd640 10 monclient: _finish_hunting 0 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.969+0000 7f402effd640 1 monclient: found mon.noname-b 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.969+0000 7f402effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.970+0000 7f402effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:44.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.970+0000 7f402effd640 1 -- 172.21.15.69:0/3128389520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020009400 con 0x7f40300a8b20 2024-01-21T04:33:44.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.972+0000 7f402effd640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40300a8b20 0x7f40301448a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f40300b89d0 tx=0x7f40200046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:44.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.972+0000 7f4017fff640 10 client.?.objecter ms_handle_connect 0x7f40300a8b20 2024-01-21T04:33:44.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.972+0000 7f4017fff640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:44.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 1 -- 172.21.15.69:0/3128389520 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402003c880 con 0x7f40300a8b20 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 10 monclient: dump: 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:44.979 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 10 monclient: _finish_auth 0 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 10 monclient: _check_auth_tickets 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 1 -- 172.21.15.69:0/3128389520 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402003c9e0 con 0x7f40300a8b20 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4035a07640 5 monclient: authenticate success, global_id 4279 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4035a07640 10 monclient: _renew_subs 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4035a07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4035a07640 1 -- 172.21.15.69:0/3128389520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4030145170 con 0x7f40300a8b20 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 1 -- 172.21.15.69:0/3128389520 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402003cd90 con 0x7f40300a8b20 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:44.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4017fff640 10 monclient: dump: 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f402d7fa640 4 set_mon_vals no callback set 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4035a07640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4035a07640 10 monclient: _renew_subs 2024-01-21T04:33:44.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4035a07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:44.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4035a07640 1 -- 172.21.15.69:0/3128389520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40301daac0 con 0x7f40300a8b20 2024-01-21T04:33:44.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.975+0000 7f4035a07640 1 librados: init done 2024-01-21T04:33:44.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.976+0000 7f4035a07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:44.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.976+0000 7f4035a07640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:44.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.976+0000 7f4035a07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:44.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:44.976+0000 7f4035a07640 1 -- 172.21.15.69:0/3128389520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40300a9170 con 0x7f40300a8b20 2024-01-21T04:33:45.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.220+0000 7f4017fff640 1 -- 172.21.15.69:0/3128389520 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4020039040 con 0x7f40300a8b20 2024-01-21T04:33:45.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.220+0000 7f4017fff640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f401c06c8b0 0x7f401c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:45.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.220+0000 7f402e7fc640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f401c06c8b0 0x7f401c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:45.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.220+0000 7f402e7fc640 10 monclient: get_auth_request con 0x7f401c06c8b0 auth_method 0 2024-01-21T04:33:45.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.221+0000 7f4017fff640 10 client.4279.objecter ms_handle_connect 0x7f401c06c8b0 2024-01-21T04:33:45.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.221+0000 7f402e7fc640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f401c06c8b0 0x7f401c06ed70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4024007fd0 tx=0x7f4024009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:45.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.222+0000 7f4017fff640 1 -- 172.21.15.69:0/3128389520 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4020079dc0 con 0x7f40300a8b20 2024-01-21T04:33:45.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.222+0000 7f4017fff640 10 client.4279.objecter ms_dispatch 0x7f4030000b90 osd_map(22..22 src has 1..22) v4 2024-01-21T04:33:45.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.222+0000 7f4017fff640 3 client.4279.objecter handle_osd_map got epochs [22,22] > 0 2024-01-21T04:33:45.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.222+0000 7f4017fff640 3 client.4279.objecter handle_osd_map decoding full epoch 22 2024-01-21T04:33:45.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.222+0000 7f4017fff640 20 client.4279.objecter dump_active .. 0 homeless 2024-01-21T04:33:45.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.234+0000 7f4017fff640 1 -- 172.21.15.69:0/3128389520 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4020076100 con 0x7f40300a8b20 2024-01-21T04:33:45.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.234+0000 7f4017fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:45.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.234+0000 7f4017fff640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:45.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.444+0000 7f4035a07640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-21T04:33:45.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.444+0000 7f4035a07640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-21T04:33:45.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.444+0000 7f4035a07640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:45.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:45.444+0000 7f4035a07640 1 -- 172.21.15.69:0/3128389520 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f40300a9170 con 0x7f40300a8b20 2024-01-21T04:33:46.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.455+0000 7f4017fff640 1 -- 172.21.15.69:0/3128389520 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v23) v1 ==== 109+0+0 (secure 0 0 0) 0x7f40300a9170 con 0x7f40300a8b20 2024-01-21T04:33:46.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.455+0000 7f4017fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-01-21T04:33:46.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.455+0000 7f4017fff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-01-21T04:33:46.456 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'rbd' created 2024-01-21T04:33:46.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.456+0000 7f4035a07640 10 librados: watch_flush enter 2024-01-21T04:33:46.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 10 librados: watch_flush exit 2024-01-21T04:33:46.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 20 client.4279.objecter shutdown clearing up homeless session... 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 10 client.4279.objecter successfully canceled tick 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 -- 172.21.15.69:0/3128389520 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f401c06c8b0 msgr2=0x7f401c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f401c06c8b0 0x7f401c06ed70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4024007fd0 tx=0x7f4024009040 comp rx=0 tx=0).stop 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 10 monclient: shutdown 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 -- 172.21.15.69:0/3128389520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40300a8b20 msgr2=0x7f40301448a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40300a8b20 0x7f40301448a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f40300b89d0 tx=0x7f40200046c0 comp rx=0 tx=0).stop 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 -- 172.21.15.69:0/3128389520 shutdown_connections 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f401c06c8b0 0x7f401c06ed70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40300b2310 0x7f4030140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f40300a9d10 0x7f4030135ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 --2- 172.21.15.69:0/3128389520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f40300a8b20 0x7f40301448a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 -- 172.21.15.69:0/3128389520 >> 172.21.15.69:0/3128389520 conn(0x7f4030066fb0 msgr2=0x7f403013ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:46.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 -- 172.21.15.69:0/3128389520 shutdown_connections 2024-01-21T04:33:46.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 -- 172.21.15.69:0/3128389520 wait complete. 2024-01-21T04:33:46.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.457+0000 7f4035a07640 1 librados: shutdown 2024-01-21T04:33:46.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-21T04:33:46.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.570+0000 7fd6dc33a640 10 monclient: get_monmap_and_config 2024-01-21T04:33:46.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.570+0000 7fd6dc33a640 10 monclient: build_initial_monmap 2024-01-21T04:33:46.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.570+0000 7fd6dc33a640 10 monclient: monmap: 2024-01-21T04:33:46.571 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:46.571 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:46.571 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:46.571283+0000 2024-01-21T04:33:46.571 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:46.571283+0000 2024-01-21T04:33:46.571 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:46.571 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:46.571 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:46.572 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:46.572 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:46.572 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:46.572 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:46.572 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:46.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.570+0000 7fd6dc33a640 1 Processor -- start 2024-01-21T04:33:46.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.570+0000 7fd6dc33a640 1 -- start start 2024-01-21T04:33:46.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.570+0000 7fd6dc33a640 10 monclient: init 2024-01-21T04:33:46.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:46.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-01-21T04:33:46.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd6d40a8b20 0x7fd6d4145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:46.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 10 monclient(hunting): picked mon.noname-a con 0x7fd6d40a8b20 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:46.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6d40a9d10 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:46.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6d40b2310 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:46.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d41456f0 con 0x7fd6d40a9d10 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d40a98c0 con 0x7fd6d40b2310 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d41460c0 con 0x7fd6d40a8b20 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.571+0000 7fd6dc33a640 10 monclient(hunting): authenticate will time out at 1384.912843s 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.572+0000 7fd6da8d7640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fd6d40b2310 0x7fd6d414c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:53906/0 (socket says 172.21.15.69:53906) 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.572+0000 7fd6da8d7640 1 -- 172.21.15.69:0/716086245 learned_addr learned my addr 172.21.15.69:0/716086245 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.572+0000 7fd6da0d6640 1 --2- 172.21.15.69:0/716086245 >> v2:172.21.15.69:3300/0 conn(0x7fd6d40a8b20 0x7fd6d4145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.572+0000 7fd6da0d6640 10 monclient(hunting): get_auth_request con 0x7fd6d40a8b20 auth_method 0 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.572+0000 7fd6da0d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.572+0000 7fd6da0d6640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.572+0000 7fd6da0d6640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.573+0000 7fd6da0d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.573+0000 7fd6da0d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:46.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.573+0000 7fd6da0d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.575+0000 7fd6d90d4640 1 -- 172.21.15.69:0/716086245 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706917366 0 0) 0x7fd6d40a98c0 con 0x7fd6d40b2310 2024-01-21T04:33:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.575+0000 7fd6d90d4640 10 monclient(hunting): _negotiate 2024-01-21T04:33:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.575+0000 7fd6d90d4640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.575+0000 7fd6d90d4640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.575+0000 7fd6d90d4640 10 monclient(hunting): my global_id is 4282 2024-01-21T04:33:46.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.575+0000 7fd6d90d4640 1 -- 172.21.15.69:0/716086245 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c0003520 con 0x7fd6d40b2310 2024-01-21T04:33:46.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.576+0000 7fd6da0d6640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2024-01-21T04:33:46.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.576+0000 7fd6da0d6640 1 -- 172.21.15.69:0/716086245 >> v1:172.21.15.144:6789/0 conn(0x7fd6d40b2310 legacy=0x7fd6d414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:46.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.576+0000 7fd6da0d6640 1 -- 172.21.15.69:0/716086245 >> v1:172.21.15.69:6789/0 conn(0x7fd6d40a9d10 legacy=0x7fd6d413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:46.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.576+0000 7fd6da0d6640 10 monclient: _finish_hunting 0 2024-01-21T04:33:46.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.576+0000 7fd6da0d6640 1 monclient: found mon.noname-a 2024-01-21T04:33:46.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.576+0000 7fd6da0d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:46.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.576+0000 7fd6da0d6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:46.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.576+0000 7fd6da0d6640 1 -- 172.21.15.69:0/716086245 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d4146200 con 0x7fd6d40a8b20 2024-01-21T04:33:46.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.579+0000 7fd6da0d6640 1 --2- 172.21.15.69:0/716086245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40a8b20 0x7fd6d4145b80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0097a0 tx=0x7fd6cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83eb69b899284bb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:46.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 1 -- 172.21.15.69:0/716086245 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc031830 con 0x7fd6d40a8b20 2024-01-21T04:33:46.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:46.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:46.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 10 monclient: dump: 2024-01-21T04:33:46.585 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:46.585 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:46.585 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:46.585 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 10 monclient: _finish_auth 0 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 10 monclient: _check_auth_tickets 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 1 -- 172.21.15.69:0/716086245 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc031990 con 0x7fd6d40a8b20 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 1 -- 172.21.15.69:0/716086245 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc031d20 con 0x7fd6d40a8b20 2024-01-21T04:33:46.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6d90d4640 10 monclient: dump: 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6dc33a640 5 monclient: authenticate success, global_id 4272 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6dc33a640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:46.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.583+0000 7fd6dc33a640 4 set_mon_vals no callback set 2024-01-21T04:33:46.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.584+0000 7fd6dc33a640 10 monclient: shutdown 2024-01-21T04:33:46.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.584+0000 7fd6dc33a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:46.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.584+0000 7fd6dc33a640 1 -- 172.21.15.69:0/716086245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40a8b20 msgr2=0x7fd6d4145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:46.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.584+0000 7fd6dc33a640 1 --2- 172.21.15.69:0/716086245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40a8b20 0x7fd6d4145b80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0097a0 tx=0x7fd6cc02b8d0 comp rx=0 tx=0).stop 2024-01-21T04:33:46.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.584+0000 7fd6dc33a640 1 -- 172.21.15.69:0/716086245 shutdown_connections 2024-01-21T04:33:46.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.584+0000 7fd6dc33a640 1 --2- 172.21.15.69:0/716086245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40a8b20 0x7fd6d4145b80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:46.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.584+0000 7fd6dc33a640 1 -- 172.21.15.69:0/716086245 >> 172.21.15.69:0/716086245 conn(0x7fd6d4066fb0 msgr2=0x7fd6d40b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:46.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.584+0000 7fd6dc33a640 1 -- 172.21.15.69:0/716086245 shutdown_connections 2024-01-21T04:33:46.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.584+0000 7fd6dc33a640 1 -- 172.21.15.69:0/716086245 wait complete. 2024-01-21T04:33:46.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.585+0000 7fd6dc33a640 10 monclient: build_initial_monmap 2024-01-21T04:33:46.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.585+0000 7fd6dc33a640 10 monclient: monmap: 2024-01-21T04:33:46.591 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.585+0000 7fd6dc33a640 1 librados: starting msgr at 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.585+0000 7fd6dc33a640 1 librados: starting objecter 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.585+0000 7fd6dc33a640 1 Processor -- start 2024-01-21T04:33:46.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.585+0000 7fd6dc33a640 1 -- start start 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.585+0000 7fd6dc33a640 1 librados: setting wanted keys 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.585+0000 7fd6dc33a640 1 librados: calling monclient init 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.585+0000 7fd6dc33a640 10 monclient: init 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d40a8b20 0x7fd6d41420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 10 monclient(hunting): picked mon.noname-b con 0x7fd6d40a8b20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40a9d10 0x7fd6d4143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 10 monclient(hunting): picked mon.noname-a con 0x7fd6d40a9d10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6d40b2310 0x7fd6d41d74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 10 monclient(hunting): picked mon.noname-c con 0x7fd6d40b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d40ab9a0 con 0x7fd6d40a9d10 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:46.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d4141bb0 con 0x7fd6d40a8b20 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d41441d0 con 0x7fd6d40b2310 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6dc33a640 10 monclient(hunting): authenticate will time out at 1384.927463s 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6d98d5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40a9d10 0x7fd6d4143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6da8d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6d40b2310 0x7fd6d41d74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.586+0000 7fd6da0d6640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d40a8b20 0x7fd6d41420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6d98d5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40a9d10 0x7fd6d4143c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53132/0 (socket says 172.21.15.69:53132) 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6d98d5640 1 -- 172.21.15.69:0/776199077 learned_addr learned my addr 172.21.15.69:0/776199077 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6d98d5640 10 monclient(hunting): get_auth_request con 0x7fd6d40a9d10 auth_method 0 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6d98d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6d98d5640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6d98d5640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da8d7640 10 monclient(hunting): get_auth_request con 0x7fd6d40b2310 auth_method 0 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da8d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da8d7640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da8d7640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da0d6640 10 monclient(hunting): get_auth_request con 0x7fd6d40a8b20 auth_method 0 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da0d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da0d6640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da0d6640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da8d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da8d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da8d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6d98d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6d98d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6d98d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da0d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da0d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.587+0000 7fd6da0d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.590+0000 7fd6d98d5640 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.590+0000 7fd6d98d5640 1 -- 172.21.15.69:0/776199077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6d40b2310 msgr2=0x7fd6d41d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.590+0000 7fd6d98d5640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6d40b2310 0x7fd6d41d74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.590+0000 7fd6d98d5640 1 -- 172.21.15.69:0/776199077 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d40a8b20 msgr2=0x7fd6d41420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.590+0000 7fd6d98d5640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d40a8b20 0x7fd6d41420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.590+0000 7fd6d98d5640 10 monclient: _finish_hunting 0 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.590+0000 7fd6d98d5640 1 monclient: found mon.noname-a 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.590+0000 7fd6d98d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.590+0000 7fd6d98d5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.590+0000 7fd6d98d5640 1 -- 172.21.15.69:0/776199077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc009400 con 0x7fd6d40a9d10 2024-01-21T04:33:46.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.592+0000 7fd6d98d5640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40a9d10 0x7fd6d4143c90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd6c400c440 tx=0x7fd6c4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:46.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.592+0000 7fd6caffd640 10 client.?.objecter ms_handle_connect 0x7fd6d40a9d10 2024-01-21T04:33:46.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.592+0000 7fd6caffd640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:46.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 1 -- 172.21.15.69:0/776199077 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40052d0 con 0x7fd6d40a9d10 2024-01-21T04:33:46.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:46.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:46.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 10 monclient: dump: 2024-01-21T04:33:46.598 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 10 monclient: _finish_auth 0 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 1 -- 172.21.15.69:0/776199077 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c4005430 con 0x7fd6d40a9d10 2024-01-21T04:33:46.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6dc33a640 5 monclient: authenticate success, global_id 4278 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6dc33a640 10 monclient: _renew_subs 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6dc33a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6dc33a640 1 -- 172.21.15.69:0/776199077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d4141810 con 0x7fd6d40a9d10 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 1 -- 172.21.15.69:0/776199077 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4007260 con 0x7fd6d40a9d10 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6caffd640 10 monclient: dump: 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:46.600 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6d88d3640 4 set_mon_vals no callback set 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6dc33a640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6dc33a640 10 monclient: _renew_subs 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6dc33a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6dc33a640 1 -- 172.21.15.69:0/776199077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d41d7bd0 con 0x7fd6d40a9d10 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.596+0000 7fd6dc33a640 1 librados: init done 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.597+0000 7fd6dc33a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:46.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.597+0000 7fd6dc33a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:46.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.597+0000 7fd6dc33a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:46.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.597+0000 7fd6dc33a640 1 -- 172.21.15.69:0/776199077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d404e970 con 0x7fd6d40a9d10 2024-01-21T04:33:46.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.621+0000 7fd6caffd640 1 -- 172.21.15.69:0/776199077 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fd6c40073c0 con 0x7fd6d40a9d10 2024-01-21T04:33:46.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.622+0000 7fd6caffd640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fd6b806c8b0 0x7fd6b806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:46.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.622+0000 7fd6da0d6640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fd6b806c8b0 0x7fd6b806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:46.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.622+0000 7fd6da0d6640 10 monclient: get_auth_request con 0x7fd6b806c8b0 auth_method 0 2024-01-21T04:33:46.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.623+0000 7fd6caffd640 10 client.4278.objecter ms_handle_connect 0x7fd6b806c8b0 2024-01-21T04:33:46.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.623+0000 7fd6da0d6640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fd6b806c8b0 0x7fd6b806ed70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc036000 tx=0x7fd6cc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.624+0000 7fd6caffd640 1 -- 172.21.15.69:0/776199077 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5199+0+0 (secure 0 0 0) 0x7fd6c4057e40 con 0x7fd6d40a9d10 2024-01-21T04:33:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.624+0000 7fd6caffd640 10 client.4278.objecter ms_dispatch 0x7fd6d4000b90 osd_map(23..23 src has 1..23) v4 2024-01-21T04:33:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.624+0000 7fd6caffd640 3 client.4278.objecter handle_osd_map got epochs [23,23] > 0 2024-01-21T04:33:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.624+0000 7fd6caffd640 3 client.4278.objecter handle_osd_map decoding full epoch 23 2024-01-21T04:33:46.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.624+0000 7fd6caffd640 20 client.4278.objecter dump_active .. 0 homeless 2024-01-21T04:33:46.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.871+0000 7fd6caffd640 1 -- 172.21.15.69:0/776199077 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fd6c4054180 con 0x7fd6d40a9d10 2024-01-21T04:33:46.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.871+0000 7fd6caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:46.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:46.871+0000 7fd6caffd640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:47.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.072+0000 7fd6dc33a640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-21T04:33:47.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.072+0000 7fd6dc33a640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-21T04:33:47.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.072+0000 7fd6dc33a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:47.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.072+0000 7fd6dc33a640 1 -- 172.21.15.69:0/776199077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd6d4072860 con 0x7fd6d40a9d10 2024-01-21T04:33:47.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.404+0000 7fd6caffd640 1 -- 172.21.15.69:0/776199077 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v24) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd6c40200c0 con 0x7fd6d40a9d10 2024-01-21T04:33:47.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.404+0000 7fd6caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-01-21T04:33:47.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.404+0000 7fd6caffd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-01-21T04:33:47.405 INFO:teuthology.orchestra.run.smithi069.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-21T04:33:47.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 10 librados: watch_flush enter 2024-01-21T04:33:47.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 10 librados: watch_flush exit 2024-01-21T04:33:47.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 20 client.4278.objecter shutdown clearing up homeless session... 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 10 client.4278.objecter successfully canceled tick 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 -- 172.21.15.69:0/776199077 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fd6b806c8b0 msgr2=0x7fd6b806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fd6b806c8b0 0x7fd6b806ed70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc036000 tx=0x7fd6cc03c040 comp rx=0 tx=0).stop 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 10 monclient: shutdown 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 -- 172.21.15.69:0/776199077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40a9d10 msgr2=0x7fd6d4143c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40a9d10 0x7fd6d4143c90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd6c400c440 tx=0x7fd6c4001e70 comp rx=0 tx=0).stop 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 -- 172.21.15.69:0/776199077 shutdown_connections 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fd6b806c8b0 0x7fd6b806ed70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fd6d40b2310 0x7fd6d41d74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40a9d10 0x7fd6d4143c90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 --2- 172.21.15.69:0/776199077 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6d40a8b20 0x7fd6d41420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:47.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 -- 172.21.15.69:0/776199077 >> 172.21.15.69:0/776199077 conn(0x7fd6d4066fb0 msgr2=0x7fd6d413ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:47.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 -- 172.21.15.69:0/776199077 shutdown_connections 2024-01-21T04:33:47.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 -- 172.21.15.69:0/776199077 wait complete. 2024-01-21T04:33:47.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.406+0000 7fd6dc33a640 1 librados: shutdown 2024-01-21T04:33:47.418 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-21T04:33:47.419 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-21T04:33:47.419 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-21T04:33:47.419 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-21T04:33:47.419 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-21T04:33:47.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.526+0000 7efe36fe2640 10 monclient: get_monmap_and_config 2024-01-21T04:33:47.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.526+0000 7efe36fe2640 10 monclient: build_initial_monmap 2024-01-21T04:33:47.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.526+0000 7efe36fe2640 10 monclient: monmap: 2024-01-21T04:33:47.527 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:47.527 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:47.527 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:47.527342+0000 2024-01-21T04:33:47.527 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:47.527342+0000 2024-01-21T04:33:47.527 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:47.527 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:47.527 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:47.528 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:47.528 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:47.528 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:47.528 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:47.528 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:47.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 1 Processor -- start 2024-01-21T04:33:47.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 1 -- start start 2024-01-21T04:33:47.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 10 monclient: init 2024-01-21T04:33:47.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:47.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-01-21T04:33:47.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efe300a8a70 0x7efe3013efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:47.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 10 monclient(hunting): picked mon.noname-c con 0x7efe300a8a70 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:47.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe300a8e50 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:47.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe30142b60 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:47.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe300aa690 con 0x7efe300a8e50 2024-01-21T04:33:47.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe300aa970 con 0x7efe30142b60 2024-01-21T04:33:47.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:47.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe300a9f90 con 0x7efe300a8a70 2024-01-21T04:33:47.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:47.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.527+0000 7efe36fe2640 10 monclient(hunting): authenticate will time out at 1385.869015s 2024-01-21T04:33:47.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.555+0000 7efe34d7e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7efe300a8a70 0x7efe3013efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.556+0000 7efe3557f640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7efe30142b60 0x7efe3014c8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:53922/0 (socket says 172.21.15.69:53922) 2024-01-21T04:33:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.556+0000 7efe3557f640 1 -- 172.21.15.69:0/1753142298 learned_addr learned my addr 172.21.15.69:0/1753142298 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.556+0000 7efe34d7e640 10 monclient(hunting): get_auth_request con 0x7efe300a8a70 auth_method 0 2024-01-21T04:33:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.556+0000 7efe34d7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.556+0000 7efe34d7e640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.556+0000 7efe34d7e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.557+0000 7efe34d7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.557+0000 7efe34d7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.557+0000 7efe34d7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.562+0000 7efe34d7e640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2024-01-21T04:33:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.562+0000 7efe34d7e640 1 -- 172.21.15.69:0/1753142298 >> v1:172.21.15.144:6789/0 conn(0x7efe30142b60 legacy=0x7efe3014c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.562+0000 7efe34d7e640 1 -- 172.21.15.69:0/1753142298 >> v1:172.21.15.69:6789/0 conn(0x7efe300a8e50 legacy=0x7efe30142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.562+0000 7efe34d7e640 10 monclient: _finish_hunting 0 2024-01-21T04:33:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.562+0000 7efe34d7e640 1 monclient: found mon.noname-c 2024-01-21T04:33:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.562+0000 7efe34d7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.562+0000 7efe34d7e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.562+0000 7efe34d7e640 1 -- 172.21.15.69:0/1753142298 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe300aac50 con 0x7efe300a8a70 2024-01-21T04:33:47.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.566+0000 7efe34d7e640 1 --2- 172.21.15.69:0/1753142298 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe300a8a70 0x7efe3013efc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efe240097a0 tx=0x7efe2402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a17f7a22ea6cd6d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:47.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 1 -- 172.21.15.69:0/1753142298 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2402e070 con 0x7efe300a8a70 2024-01-21T04:33:47.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:47.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:47.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 10 monclient: dump: 2024-01-21T04:33:47.572 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:47.572 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:47.572 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:47.572 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 1 -- 172.21.15.69:0/1753142298 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe240315a0 con 0x7efe300a8a70 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe36fe2640 5 monclient: authenticate success, global_id 4288 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe36fe2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe36fe2640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:47.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 1 -- 172.21.15.69:0/1753142298 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe24031930 con 0x7efe300a8a70 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe36fe2640 4 set_mon_vals no callback set 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.571+0000 7efe2f7fe640 10 monclient: dump: 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:47.574 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:47.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.572+0000 7efe36fe2640 10 monclient: shutdown 2024-01-21T04:33:47.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.572+0000 7efe36fe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.572+0000 7efe36fe2640 1 -- 172.21.15.69:0/1753142298 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe300a8a70 msgr2=0x7efe3013efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.572+0000 7efe36fe2640 1 --2- 172.21.15.69:0/1753142298 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe300a8a70 0x7efe3013efc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efe240097a0 tx=0x7efe2402b840 comp rx=0 tx=0).stop 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.572+0000 7efe36fe2640 1 -- 172.21.15.69:0/1753142298 shutdown_connections 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.572+0000 7efe36fe2640 1 --2- 172.21.15.69:0/1753142298 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe300a8a70 0x7efe3013efc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.572+0000 7efe36fe2640 1 -- 172.21.15.69:0/1753142298 >> 172.21.15.69:0/1753142298 conn(0x7efe30066f00 msgr2=0x7efe300b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.572+0000 7efe36fe2640 1 -- 172.21.15.69:0/1753142298 shutdown_connections 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.572+0000 7efe36fe2640 1 -- 172.21.15.69:0/1753142298 wait complete. 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.573+0000 7efe36fe2640 10 monclient: build_initial_monmap 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.573+0000 7efe36fe2640 10 monclient: monmap: 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:47.578 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.573+0000 7efe36fe2640 1 librados: starting msgr at 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.573+0000 7efe36fe2640 1 librados: starting objecter 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.573+0000 7efe36fe2640 1 Processor -- start 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.573+0000 7efe36fe2640 1 -- start start 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.573+0000 7efe36fe2640 1 librados: setting wanted keys 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.573+0000 7efe36fe2640 1 librados: calling monclient init 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.573+0000 7efe36fe2640 10 monclient: init 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:33:47.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe300a8a70 0x7efe300ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 10 monclient(hunting): picked mon.noname-a con 0x7efe300a8a70 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe300a8e50 0x7efe301d7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 10 monclient(hunting): picked mon.noname-b con 0x7efe300a8e50 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe30142b60 0x7efe301d7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 10 monclient(hunting): picked mon.noname-c con 0x7efe30142b60 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe301d7320 con 0x7efe300a8a70 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe300adce0 con 0x7efe300a8e50 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe30144320 con 0x7efe30142b60 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:47.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.574+0000 7efe36fe2640 10 monclient(hunting): authenticate will time out at 1385.915838s 2024-01-21T04:33:47.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.575+0000 7efe3557f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe30142b60 0x7efe301d7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:47.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.575+0000 7efe3557f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe30142b60 0x7efe301d7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46764/0 (socket says 172.21.15.69:46764) 2024-01-21T04:33:47.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.575+0000 7efe3557f640 1 -- 172.21.15.69:0/3484018236 learned_addr learned my addr 172.21.15.69:0/3484018236 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:47.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.575+0000 7efe3557f640 10 monclient(hunting): get_auth_request con 0x7efe30142b60 auth_method 0 2024-01-21T04:33:47.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.575+0000 7efe3557f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:47.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.575+0000 7efe3557f640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:47.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.575+0000 7efe3557f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:47.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.576+0000 7efe3557f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:47.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.576+0000 7efe3557f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:47.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.576+0000 7efe3557f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:47.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.577+0000 7efe2ffff640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe300a8e50 0x7efe301d7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:47.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.577+0000 7efe34d7e640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe300a8a70 0x7efe300ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:47.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe2ffff640 10 monclient(hunting): get_auth_request con 0x7efe300a8e50 auth_method 0 2024-01-21T04:33:47.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:47.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe2ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:47.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:47.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe34d7e640 10 monclient(hunting): get_auth_request con 0x7efe300a8a70 auth_method 0 2024-01-21T04:33:47.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe34d7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:47.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe34d7e640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:47.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe34d7e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe34d7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe34d7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe34d7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe3557f640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe3557f640 1 -- 172.21.15.69:0/3484018236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe300a8e50 msgr2=0x7efe301d7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe3557f640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe300a8e50 0x7efe301d7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe3557f640 1 -- 172.21.15.69:0/3484018236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe300a8a70 msgr2=0x7efe300ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe3557f640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe300a8a70 0x7efe300ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe3557f640 10 monclient: _finish_hunting 0 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.578+0000 7efe3557f640 1 monclient: found mon.noname-c 2024-01-21T04:33:47.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.579+0000 7efe3557f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.579+0000 7efe3557f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:33:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.579+0000 7efe3557f640 1 -- 172.21.15.69:0/3484018236 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe24009400 con 0x7efe30142b60 2024-01-21T04:33:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.581+0000 7efe3557f640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe30142b60 0x7efe301d7cb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efe2800ca80 tx=0x7efe2800cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.581+0000 7efe2d7fa640 10 client.?.objecter ms_handle_connect 0x7efe30142b60 2024-01-21T04:33:47.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.581+0000 7efe2d7fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:47.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 1 -- 172.21.15.69:0/3484018236 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2800ac30 con 0x7efe30142b60 2024-01-21T04:33:47.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:47.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:47.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 10 monclient: dump: 2024-01-21T04:33:47.587 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:47.587 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:47.587 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:47.587 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 10 monclient: _finish_auth 0 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 10 monclient: _check_auth_tickets 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 1 -- 172.21.15.69:0/3484018236 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2800ad90 con 0x7efe30142b60 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 1 -- 172.21.15.69:0/3484018236 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe28012600 con 0x7efe30142b60 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:47.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2d7fa640 10 monclient: dump: 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe36fe2640 5 monclient: authenticate success, global_id 4256 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe36fe2640 10 monclient: _renew_subs 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe36fe2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe36fe2640 1 -- 172.21.15.69:0/3484018236 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe300acbb0 con 0x7efe30142b60 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe2effd640 4 set_mon_vals no callback set 2024-01-21T04:33:47.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe36fe2640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe36fe2640 10 monclient: _renew_subs 2024-01-21T04:33:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe36fe2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe36fe2640 1 -- 172.21.15.69:0/3484018236 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe300abb40 con 0x7efe30142b60 2024-01-21T04:33:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.585+0000 7efe36fe2640 1 librados: init done 2024-01-21T04:33:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.586+0000 7efe36fe2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.586+0000 7efe36fe2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.586+0000 7efe36fe2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:47.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.586+0000 7efe36fe2640 1 -- 172.21.15.69:0/3484018236 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe3013f4e0 con 0x7efe30142b60 2024-01-21T04:33:47.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.605+0000 7efe2d7fa640 1 -- 172.21.15.69:0/3484018236 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7efe28007070 con 0x7efe30142b60 2024-01-21T04:33:47.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.605+0000 7efe2d7fa640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7efe1406c860 0x7efe1406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:47.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.605+0000 7efe34d7e640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7efe1406c860 0x7efe1406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:47.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.606+0000 7efe34d7e640 10 monclient: get_auth_request con 0x7efe1406c860 auth_method 0 2024-01-21T04:33:47.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.606+0000 7efe2d7fa640 10 client.4256.objecter ms_handle_connect 0x7efe1406c860 2024-01-21T04:33:47.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.606+0000 7efe34d7e640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7efe1406c860 0x7efe1406ed20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efe24036000 tx=0x7efe2403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.607+0000 7efe2d7fa640 1 -- 172.21.15.69:0/3484018236 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5210+0+0 (secure 0 0 0) 0x7efe280583d0 con 0x7efe30142b60 2024-01-21T04:33:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.607+0000 7efe2d7fa640 10 client.4256.objecter ms_dispatch 0x7efe30000b90 osd_map(24..24 src has 1..24) v4 2024-01-21T04:33:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.607+0000 7efe2d7fa640 3 client.4256.objecter handle_osd_map got epochs [24,24] > 0 2024-01-21T04:33:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.607+0000 7efe2d7fa640 3 client.4256.objecter handle_osd_map decoding full epoch 24 2024-01-21T04:33:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.607+0000 7efe2d7fa640 20 client.4256.objecter dump_active .. 0 homeless 2024-01-21T04:33:47.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.878+0000 7efe2d7fa640 1 -- 172.21.15.69:0/3484018236 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7efe28054710 con 0x7efe30142b60 2024-01-21T04:33:47.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.878+0000 7efe2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:47.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:47.878+0000 7efe2d7fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.148+0000 7efe36fe2640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-01-21T04:33:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.148+0000 7efe36fe2640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-01-21T04:33:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.148+0000 7efe36fe2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.148+0000 7efe36fe2640 1 -- 172.21.15.69:0/3484018236 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7efe301d7490 con 0x7efe30142b60 2024-01-21T04:33:48.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.292+0000 7efe2d7fa640 1 -- 172.21.15.69:0/3484018236 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170775 (secure 0 0 0) 0x7efe301d7490 con 0x7efe30142b60 2024-01-21T04:33:48.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.292+0000 7efe2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-01-21T04:33:48.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.292+0000 7efe2d7fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:48.296 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 10 librados: watch_flush enter 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 10 librados: watch_flush exit 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 20 client.4256.objecter shutdown clearing up homeless session... 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 10 client.4256.objecter successfully canceled tick 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 -- 172.21.15.69:0/3484018236 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7efe1406c860 msgr2=0x7efe1406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7efe1406c860 0x7efe1406ed20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efe24036000 tx=0x7efe2403d040 comp rx=0 tx=0).stop 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 10 monclient: shutdown 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 -- 172.21.15.69:0/3484018236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe30142b60 msgr2=0x7efe301d7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe30142b60 0x7efe301d7cb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efe2800ca80 tx=0x7efe2800cf40 comp rx=0 tx=0).stop 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 -- 172.21.15.69:0/3484018236 shutdown_connections 2024-01-21T04:33:48.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7efe1406c860 0x7efe1406ed20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:48.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7efe30142b60 0x7efe301d7cb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:48.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe300a8e50 0x7efe301d7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:48.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 --2- 172.21.15.69:0/3484018236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe300a8a70 0x7efe300ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:48.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 -- 172.21.15.69:0/3484018236 >> 172.21.15.69:0/3484018236 conn(0x7efe30066f00 msgr2=0x7efe30139e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:48.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 -- 172.21.15.69:0/3484018236 shutdown_connections 2024-01-21T04:33:48.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.295+0000 7efe36fe2640 1 -- 172.21.15.69:0/3484018236 wait complete. 2024-01-21T04:33:48.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.296+0000 7efe36fe2640 1 librados: shutdown 2024-01-21T04:33:48.307 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":5,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1966288294},{"type":"v1","addr":"172.21.15.144:6801","nonce":1966288294}]},"active_addr":"172.21.15.144:6801/1966288294","active_change":"2024-01-21T04:32:49.114956+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[{"gid":4111,"name":"y","mgr_features":4540701272860131327,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":916466021}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3186208301}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":3747384865}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.144:0","nonce":2873879626}]}]} 2024-01-21T04:33:48.313 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-21T04:33:48.313 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-21T04:33:48.313 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-21T04:33:48.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.417+0000 7f25eee56640 10 monclient: get_monmap_and_config 2024-01-21T04:33:48.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.417+0000 7f25eee56640 10 monclient: build_initial_monmap 2024-01-21T04:33:48.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.417+0000 7f25eee56640 10 monclient: monmap: 2024-01-21T04:33:48.418 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:48.418 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:48.418 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:48.418199+0000 2024-01-21T04:33:48.418 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:48.418199+0000 2024-01-21T04:33:48.419 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:48.419 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:48.419 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:48.419 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:48.419 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:48.419 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:48.419 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:48.419 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:48.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 1 Processor -- start 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 1 -- start start 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 10 monclient: init 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f25e80a8a40 0x7f25e8145b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 10 monclient(hunting): picked mon.noname-a con 0x7f25e80a8a40 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f25e81460a0 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25e813f8b0 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e80aa9b0 con 0x7f25e81460a0 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e80aac90 con 0x7f25e813f8b0 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e80aa7d0 con 0x7f25e80a8a40 2024-01-21T04:33:48.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:48.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.418+0000 7f25eee56640 10 monclient(hunting): authenticate will time out at 1386.759935s 2024-01-21T04:33:48.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.419+0000 7f25ecbf2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f25e80a8a40 0x7f25e8145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:48.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.419+0000 7f25e7fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f25e81460a0 0x7f25e813f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58838/0 (socket says 172.21.15.69:58838) 2024-01-21T04:33:48.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.419+0000 7f25e7fff640 1 -- 172.21.15.69:0/3976668340 learned_addr learned my addr 172.21.15.69:0/3976668340 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:48.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.419+0000 7f25ecbf2640 10 monclient(hunting): get_auth_request con 0x7f25e80a8a40 auth_method 0 2024-01-21T04:33:48.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.419+0000 7f25ecbf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:48.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.419+0000 7f25ecbf2640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:48.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.419+0000 7f25ecbf2640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:48.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.420+0000 7f25ecbf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:48.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.420+0000 7f25ecbf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:48.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.420+0000 7f25ecbf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:48.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.424+0000 7f25ecbf2640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-01-21T04:33:48.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.424+0000 7f25ecbf2640 1 -- 172.21.15.69:0/3976668340 >> v1:172.21.15.144:6789/0 conn(0x7f25e813f8b0 legacy=0x7f25e814c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:48.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.424+0000 7f25ecbf2640 1 -- 172.21.15.69:0/3976668340 >> v1:172.21.15.69:6789/0 conn(0x7f25e81460a0 legacy=0x7f25e813f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:48.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.424+0000 7f25ecbf2640 10 monclient: _finish_hunting 0 2024-01-21T04:33:48.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.424+0000 7f25ecbf2640 1 monclient: found mon.noname-a 2024-01-21T04:33:48.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.424+0000 7f25ecbf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:48.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.424+0000 7f25ecbf2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:48.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.424+0000 7f25ecbf2640 1 -- 172.21.15.69:0/3976668340 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e80aaf70 con 0x7f25e80a8a40 2024-01-21T04:33:48.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.430+0000 7f25ecbf2640 1 --2- 172.21.15.69:0/3976668340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25e80a8a40 0x7f25e8145b60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0097a0 tx=0x7f25dc02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4ca1b038e4d1b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:48.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 1 -- 172.21.15.69:0/3976668340 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc02f810 con 0x7f25e80a8a40 2024-01-21T04:33:48.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:48.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:48.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 10 monclient: dump: 2024-01-21T04:33:48.501 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:48.501 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:48.501 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 1 -- 172.21.15.69:0/3976668340 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25dc02f970 con 0x7f25e80a8a40 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 1 -- 172.21.15.69:0/3976668340 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc02fd00 con 0x7f25e80a8a40 2024-01-21T04:33:48.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25eee56640 5 monclient: authenticate success, global_id 4284 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25eee56640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25eee56640 4 set_mon_vals no callback set 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.500+0000 7f25e77fe640 10 monclient: dump: 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:48.503 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:48.504 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:48.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.501+0000 7f25eee56640 10 monclient: shutdown 2024-01-21T04:33:48.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.501+0000 7f25eee56640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:48.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.501+0000 7f25eee56640 1 -- 172.21.15.69:0/3976668340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25e80a8a40 msgr2=0x7f25e8145b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:48.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.501+0000 7f25eee56640 1 --2- 172.21.15.69:0/3976668340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25e80a8a40 0x7f25e8145b60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f25dc0097a0 tx=0x7f25dc02b9f0 comp rx=0 tx=0).stop 2024-01-21T04:33:48.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.501+0000 7f25eee56640 1 -- 172.21.15.69:0/3976668340 shutdown_connections 2024-01-21T04:33:48.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.501+0000 7f25eee56640 1 --2- 172.21.15.69:0/3976668340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25e80a8a40 0x7f25e8145b60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.501+0000 7f25eee56640 1 -- 172.21.15.69:0/3976668340 >> 172.21.15.69:0/3976668340 conn(0x7f25e8066f50 msgr2=0x7f25e80b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.501+0000 7f25eee56640 1 -- 172.21.15.69:0/3976668340 shutdown_connections 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.501+0000 7f25eee56640 1 -- 172.21.15.69:0/3976668340 wait complete. 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.501+0000 7f25eee56640 10 monclient: build_initial_monmap 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.501+0000 7f25eee56640 10 monclient: monmap: 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:48.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 1 librados: starting msgr at 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 1 librados: starting objecter 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 1 Processor -- start 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 1 -- start start 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 1 librados: setting wanted keys 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 1 librados: calling monclient init 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 10 monclient: init 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25e80a8a40 0x7f25e80b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 10 monclient(hunting): picked mon.noname-c con 0x7f25e80a8a40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25e813f8b0 0x7f25e8142230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.502+0000 7f25eee56640 10 monclient(hunting): picked mon.noname-b con 0x7f25e813f8b0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25eee56640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25e81460a0 0x7f25e81d71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:48.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25eee56640 10 monclient(hunting): picked mon.noname-a con 0x7f25e81460a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25eee56640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25eee56640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e8142ad0 con 0x7f25e81460a0 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25eee56640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25eee56640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e8142770 con 0x7f25e813f8b0 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25eee56640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25eee56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e8141ce0 con 0x7f25e80a8a40 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25eee56640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25eee56640 10 monclient(hunting): authenticate will time out at 1386.844247s 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25ecbf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25e80a8a40 0x7f25e80b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25e7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25e813f8b0 0x7f25e8142230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25ecbf2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25e80a8a40 0x7f25e80b78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46780/0 (socket says 172.21.15.69:46780) 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25ecbf2640 1 -- 172.21.15.69:0/2092525489 learned_addr learned my addr 172.21.15.69:0/2092525489 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25ecbf2640 10 monclient(hunting): get_auth_request con 0x7f25e80a8a40 auth_method 0 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25ecbf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:48.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25ecbf2640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:48.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.503+0000 7f25ecbf2640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:48.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.504+0000 7f25e7fff640 10 monclient(hunting): get_auth_request con 0x7f25e813f8b0 auth_method 0 2024-01-21T04:33:48.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.504+0000 7f25e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:48.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.504+0000 7f25e7fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:48.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.504+0000 7f25e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:48.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.504+0000 7f25ecbf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:48.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.504+0000 7f25ecbf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:48.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.504+0000 7f25ecbf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:48.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.504+0000 7f25e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:48.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.504+0000 7f25e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:48.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.504+0000 7f25e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:48.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.507+0000 7f25ecbf2640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-01-21T04:33:48.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.507+0000 7f25ecbf2640 1 -- 172.21.15.69:0/2092525489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25e813f8b0 msgr2=0x7f25e8142230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:48.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.507+0000 7f25ecbf2640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25e813f8b0 0x7f25e8142230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:48.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.507+0000 7f25ecbf2640 1 -- 172.21.15.69:0/2092525489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25e81460a0 msgr2=0x7f25e81d71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:48.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.507+0000 7f25ecbf2640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25e81460a0 0x7f25e81d71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:48.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.507+0000 7f25ecbf2640 10 monclient: _finish_hunting 0 2024-01-21T04:33:48.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.507+0000 7f25ecbf2640 1 monclient: found mon.noname-c 2024-01-21T04:33:48.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.507+0000 7f25ecbf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:48.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.507+0000 7f25ecbf2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:33:48.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.507+0000 7f25ecbf2640 1 -- 172.21.15.69:0/2092525489 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc009400 con 0x7f25e80a8a40 2024-01-21T04:33:48.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.509+0000 7f25ecbf2640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25e80a8a40 0x7f25e80b78f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f25dc02c000 tx=0x7f25dc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:48.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.510+0000 7f25e57fa640 10 client.?.objecter ms_handle_connect 0x7f25e80a8a40 2024-01-21T04:33:48.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.510+0000 7f25e57fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:48.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.511+0000 7f25e57fa640 1 -- 172.21.15.69:0/2092525489 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc03c040 con 0x7f25e80a8a40 2024-01-21T04:33:48.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.511+0000 7f25e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25e57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25e57fa640 10 monclient: dump: 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25e57fa640 10 monclient: _finish_auth 0 2024-01-21T04:33:48.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25e57fa640 10 monclient: _check_auth_tickets 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 5 monclient: authenticate success, global_id 4259 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 10 monclient: _renew_subs 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 1 -- 172.21.15.69:0/2092525489 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e8141f00 con 0x7f25e80a8a40 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 10 monclient: _renew_subs 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 1 -- 172.21.15.69:0/2092525489 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e81d7910 con 0x7f25e80a8a40 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 1 librados: init done 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:48.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:48.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.512+0000 7f25eee56640 1 -- 172.21.15.69:0/2092525489 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e804e970 con 0x7f25e80a8a40 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.534+0000 7f25e57fa640 1 -- 172.21.15.69:0/2092525489 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25dc0379d0 con 0x7f25e80a8a40 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.534+0000 7f25e57fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.534+0000 7f25e57fa640 1 -- 172.21.15.69:0/2092525489 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc037d40 con 0x7f25e80a8a40 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.534+0000 7f25e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.534+0000 7f25e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.534+0000 7f25e57fa640 10 monclient: dump: 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:48.535 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:48.536 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:48.536 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:48.536 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:48.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.534+0000 7f25e6ffd640 4 set_mon_vals no callback set 2024-01-21T04:33:48.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.535+0000 7f25e57fa640 1 -- 172.21.15.69:0/2092525489 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f25dc04b3b0 con 0x7f25e80a8a40 2024-01-21T04:33:48.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.535+0000 7f25e57fa640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f25cc06c860 0x7f25cc06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:48.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.536+0000 7f25e7fff640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f25cc06c860 0x7f25cc06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:48.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.536+0000 7f25e7fff640 10 monclient: get_auth_request con 0x7f25cc06c860 auth_method 0 2024-01-21T04:33:48.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.536+0000 7f25e7fff640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f25cc06c860 0x7f25cc06ed20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f25e80abaf0 tx=0x7f25d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:48.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.536+0000 7f25e57fa640 10 client.4259.objecter ms_handle_connect 0x7f25cc06c860 2024-01-21T04:33:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.821+0000 7f25e57fa640 1 -- 172.21.15.69:0/2092525489 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f25dc07a4e0 con 0x7f25e80a8a40 2024-01-21T04:33:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.821+0000 7f25e57fa640 10 client.4259.objecter ms_dispatch 0x7f25e8000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T04:33:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.821+0000 7f25e57fa640 3 client.4259.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T04:33:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.821+0000 7f25e57fa640 3 client.4259.objecter handle_osd_map decoding full epoch 25 2024-01-21T04:33:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.821+0000 7f25e57fa640 20 client.4259.objecter dump_active .. 0 homeless 2024-01-21T04:33:48.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.821+0000 7f25e57fa640 1 -- 172.21.15.69:0/2092525489 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f25dc076820 con 0x7f25e80a8a40 2024-01-21T04:33:48.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.821+0000 7f25e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:48.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:48.821+0000 7f25e57fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:49.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.012+0000 7f25eee56640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:49.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.012+0000 7f25eee56640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:49.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.012+0000 7f25eee56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:49.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.012+0000 7f25eee56640 1 -- 172.21.15.69:0/2092525489 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25e8072770 con 0x7f25e80a8a40 2024-01-21T04:33:49.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.109+0000 7f25e57fa640 1 -- 172.21.15.69:0/2092525489 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+14640 (secure 0 0 0) 0x7f25dc047020 con 0x7f25e80a8a40 2024-01-21T04:33:49.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.109+0000 7f25e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:49.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.109+0000 7f25e57fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:49.111 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:33:49.111 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T04:33:48.398337+0000","last_up_change":"2024-01-21T04:33:38.448164+0000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-21T04:33:29.939892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-21T04:33:45.459402+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6825","nonce":335500148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6827","nonce":335500148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6831","nonce":335500148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6829","nonce":335500148}]},"public_addr":"172.21.15.69:6825/335500148","cluster_addr":"172.21.15.69:6827/335500148","heartbeat_back_addr":"172.21.15.69:6831/335500148","heartbeat_front_addr":"172.21.15.69:6829/335500148","state":["exists","up"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6809","nonce":3868480563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6811","nonce":3868480563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6815","nonce":3868480563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6813","nonce":3868480563}]},"public_addr":"172.21.15.69:6809/3868480563","cluster_addr":"172.21.15.69:6811/3868480563","heartbeat_back_addr":"172.21.15.69:6815/3868480563","heartbeat_front_addr":"172.21.15.69:6813/3868480563","state":["exists","up"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6817","nonce":3954012353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6819","nonce":3954012353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6823","nonce":3954012353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6821","nonce":3954012353}]},"public_addr":"172.21.15.69:6817/3954012353","cluster_addr":"172.21.15.69:6819/3954012353","heartbeat_back_addr":"172.21.15.69:6823/3954012353","heartbeat_front_addr":"172.21.15.69:6821/3954012353","state":["exists","up"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6801","nonce":1086763384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6803","nonce":1086763384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6807","nonce":1086763384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6805","nonce":1086763384}]},"public_addr":"172.21.15.69:6801/1086763384","cluster_addr":"172.21.15.69:6803/1086763384","heartbeat_back_addr":"172.21.15.69:6807/1086763384","heartbeat_front_addr":"172.21.15.69:6805/1086763384","state":["exists","up"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6827","nonce":3897583001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6829","nonce":3897583001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6833","nonce":3897583001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6831","nonce":3897583001}]},"public_addr":"172.21.15.144:6827/3897583001","cluster_addr":"172.21.15.144:6829/3897583001","heartbeat_back_addr":"172.21.15.144:6833/3897583001","heartbeat_front_addr":"172.21.15.144:6831/3897583001","state":["exists","up"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6803","nonce":1782989653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6805","nonce":1782989653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6809","nonce":1782989653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6807","nonce":1782989653}]},"public_addr":"172.21.15.144:6803/1782989653","cluster_addr":"172.21.15.144:6805/1782989653","heartbeat_back_addr":"172.21.15.144:6809/1782989653","heartbeat_front_addr":"172.21.15.144:6807/1782989653","state":["exists","up"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6813","nonce":479183693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6818","nonce":479183693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6825","nonce":479183693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6822","nonce":479183693}]},"public_addr":"172.21.15.144:6813/479183693","cluster_addr":"172.21.15.144:6818/479183693","heartbeat_back_addr":"172.21.15.144:6825/479183693","heartbeat_front_addr":"172.21.15.144:6822/479183693","state":["exists","up"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6811","nonce":14156089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6815","nonce":14156089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6823","nonce":14156089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6819","nonce":14156089}]},"public_addr":"172.21.15.144:6811/14156089","cluster_addr":"172.21.15.144:6815/14156089","heartbeat_back_addr":"172.21.15.144:6823/14156089","heartbeat_front_addr":"172.21.15.144:6819/14156089","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:27.418834+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.672251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.150659+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.544731+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.167108+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:24.415555+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.272052+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.940928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T04:33:49.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.111+0000 7f25eee56640 10 librados: watch_flush enter 2024-01-21T04:33:49.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 10 librados: watch_flush exit 2024-01-21T04:33:49.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 20 client.4259.objecter shutdown clearing up homeless session... 2024-01-21T04:33:49.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 10 client.4259.objecter successfully canceled tick 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 -- 172.21.15.69:0/2092525489 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f25cc06c860 msgr2=0x7f25cc06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f25cc06c860 0x7f25cc06ed20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f25e80abaf0 tx=0x7f25d4009040 comp rx=0 tx=0).stop 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 10 monclient: shutdown 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 -- 172.21.15.69:0/2092525489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25e80a8a40 msgr2=0x7f25e80b78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25e80a8a40 0x7f25e80b78f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f25dc02c000 tx=0x7f25dc004730 comp rx=0 tx=0).stop 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 -- 172.21.15.69:0/2092525489 shutdown_connections 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f25cc06c860 0x7f25cc06ed20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25e81460a0 0x7f25e81d71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f25e813f8b0 0x7f25e8142230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 --2- 172.21.15.69:0/2092525489 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f25e80a8a40 0x7f25e80b78f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 -- 172.21.15.69:0/2092525489 >> 172.21.15.69:0/2092525489 conn(0x7f25e8066f50 msgr2=0x7f25e813b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 -- 172.21.15.69:0/2092525489 shutdown_connections 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 -- 172.21.15.69:0/2092525489 wait complete. 2024-01-21T04:33:49.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.112+0000 7f25eee56640 1 librados: shutdown 2024-01-21T04:33:49.125 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-21T04:33:49.125 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-21T04:33:49.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.223+0000 7fccbe626640 10 monclient: get_monmap_and_config 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.223+0000 7fccbe626640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.223+0000 7fccbe626640 10 monclient: monmap: 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:49.224829+0000 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:49.224829+0000 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:49.225 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:49.226 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:49.226 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.224+0000 7fccbe626640 1 Processor -- start 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.224+0000 7fccbe626640 1 -- start start 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.224+0000 7fccbe626640 10 monclient: init 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fccb80a8a40 0x7fccb8145b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 10 monclient(hunting): picked mon.noname-c con 0x7fccb80a8a40 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fccb81460a0 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fccb813f8b0 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb80aa9b0 con 0x7fccb813f8b0 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb80aac90 con 0x7fccb81460a0 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb80aa7d0 con 0x7fccb80a8a40 2024-01-21T04:33:49.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbe626640 10 monclient(hunting): authenticate will time out at 1387.566390s 2024-01-21T04:33:49.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbcbc3640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fccb813f8b0 0x7fccb814c7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58846/0 (socket says 172.21.15.69:58846) 2024-01-21T04:33:49.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.225+0000 7fccbcbc3640 1 -- 172.21.15.69:0/328562184 learned_addr learned my addr 172.21.15.69:0/328562184 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:49.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.226+0000 7fccb7fff640 1 --2- 172.21.15.69:0/328562184 >> v2:172.21.15.144:3300/0 conn(0x7fccb80a8a40 0x7fccb8145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.226+0000 7fccb7fff640 10 monclient(hunting): get_auth_request con 0x7fccb80a8a40 auth_method 0 2024-01-21T04:33:49.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.226+0000 7fccb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.226+0000 7fccb7fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.226+0000 7fccb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.227+0000 7fccb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.227+0000 7fccb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.227+0000 7fccb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.228+0000 7fccb6ffd640 1 -- 172.21.15.69:0/328562184 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082289120 0 0) 0x7fccb80aa9b0 con 0x7fccb813f8b0 2024-01-21T04:33:49.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.228+0000 7fccb6ffd640 10 monclient(hunting): _negotiate 2024-01-21T04:33:49.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.228+0000 7fccb6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.228+0000 7fccb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.228+0000 7fccb6ffd640 10 monclient(hunting): my global_id is 4290 2024-01-21T04:33:49.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.228+0000 7fccb6ffd640 1 -- 172.21.15.69:0/328562184 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca0003520 con 0x7fccb813f8b0 2024-01-21T04:33:49.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.229+0000 7fccb7fff640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-01-21T04:33:49.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.229+0000 7fccb7fff640 1 -- 172.21.15.69:0/328562184 >> v1:172.21.15.144:6789/0 conn(0x7fccb81460a0 legacy=0x7fccb813f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.229+0000 7fccb7fff640 1 -- 172.21.15.69:0/328562184 >> v1:172.21.15.69:6789/0 conn(0x7fccb813f8b0 legacy=0x7fccb814c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.229+0000 7fccb7fff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:49.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.229+0000 7fccb7fff640 1 monclient: found mon.noname-c 2024-01-21T04:33:49.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.229+0000 7fccb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:49.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.229+0000 7fccb7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:49.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.229+0000 7fccb7fff640 1 -- 172.21.15.69:0/328562184 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb80aaf70 con 0x7fccb80a8a40 2024-01-21T04:33:49.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.236+0000 7fccb7fff640 1 --2- 172.21.15.69:0/328562184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb80a8a40 0x7fccb8145b60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcca80097a0 tx=0x7fcca802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a59a343ec751cc9f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 1 -- 172.21.15.69:0/328562184 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca802f810 con 0x7fccb80a8a40 2024-01-21T04:33:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 10 monclient: dump: 2024-01-21T04:33:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.243 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 10 monclient: _finish_auth 0 2024-01-21T04:33:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 1 -- 172.21.15.69:0/328562184 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca802f970 con 0x7fccb80a8a40 2024-01-21T04:33:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 1 -- 172.21.15.69:0/328562184 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca802fd00 con 0x7fccb80a8a40 2024-01-21T04:33:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.240+0000 7fccb6ffd640 10 monclient: dump: 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 5 monclient: authenticate success, global_id 4297 2024-01-21T04:33:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 4 set_mon_vals no callback set 2024-01-21T04:33:49.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 10 monclient: shutdown 2024-01-21T04:33:49.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:49.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 1 -- 172.21.15.69:0/328562184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb80a8a40 msgr2=0x7fccb8145b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 1 --2- 172.21.15.69:0/328562184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb80a8a40 0x7fccb8145b60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcca80097a0 tx=0x7fcca802b9f0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 1 -- 172.21.15.69:0/328562184 shutdown_connections 2024-01-21T04:33:49.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 1 --2- 172.21.15.69:0/328562184 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb80a8a40 0x7fccb8145b60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 1 -- 172.21.15.69:0/328562184 >> 172.21.15.69:0/328562184 conn(0x7fccb8066f50 msgr2=0x7fccb80b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 1 -- 172.21.15.69:0/328562184 shutdown_connections 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.241+0000 7fccbe626640 1 -- 172.21.15.69:0/328562184 wait complete. 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.242+0000 7fccbe626640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.242+0000 7fccbe626640 10 monclient: monmap: 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:49.249 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.242+0000 7fccbe626640 1 librados: starting msgr at 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.242+0000 7fccbe626640 1 librados: starting objecter 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.242+0000 7fccbe626640 1 Processor -- start 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 1 -- start start 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 1 librados: setting wanted keys 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 1 librados: calling monclient init 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 10 monclient: init 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccb80a8a40 0x7fccb8136160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 10 monclient(hunting): picked mon.noname-a con 0x7fccb80a8a40 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fccb813f8b0 0x7fccb81da500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 10 monclient(hunting): picked mon.noname-c con 0x7fccb813f8b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb81460a0 0x7fccb81daa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 10 monclient(hunting): picked mon.noname-b con 0x7fccb81460a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb81daf80 con 0x7fccb80a8a40 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb8140d30 con 0x7fccb81460a0 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb80b7fa0 con 0x7fccb813f8b0 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.243+0000 7fccbe626640 10 monclient(hunting): authenticate will time out at 1387.584816s 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccb80a8a40 0x7fccb8136160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb77fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fccb813f8b0 0x7fccb81da500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccb80a8a40 0x7fccb8136160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53180/0 (socket says 172.21.15.69:53180) 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb7fff640 1 -- 172.21.15.69:0/695675796 learned_addr learned my addr 172.21.15.69:0/695675796 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb7fff640 10 monclient(hunting): get_auth_request con 0x7fccb80a8a40 auth_method 0 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb7fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccbcbc3640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb81460a0 0x7fccb81daa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb77fe640 10 monclient(hunting): get_auth_request con 0x7fccb813f8b0 auth_method 0 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb77fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccbcbc3640 10 monclient(hunting): get_auth_request con 0x7fccb81460a0 auth_method 0 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccbcbc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccbcbc3640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccbcbc3640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.244+0000 7fccb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.245+0000 7fccb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.245+0000 7fccb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.245+0000 7fccb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.245+0000 7fccbcbc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.245+0000 7fccbcbc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.245+0000 7fccbcbc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.247+0000 7fccb7fff640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.247+0000 7fccb7fff640 1 -- 172.21.15.69:0/695675796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fccb813f8b0 msgr2=0x7fccb81da500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.247+0000 7fccb7fff640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fccb813f8b0 0x7fccb81da500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.247+0000 7fccb7fff640 1 -- 172.21.15.69:0/695675796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb81460a0 msgr2=0x7fccb81daa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.247+0000 7fccb7fff640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb81460a0 0x7fccb81daa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.247+0000 7fccb7fff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:49.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.247+0000 7fccb7fff640 1 monclient: found mon.noname-a 2024-01-21T04:33:49.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.247+0000 7fccb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:49.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.247+0000 7fccb7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:49.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.247+0000 7fccb7fff640 1 -- 172.21.15.69:0/695675796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca8009400 con 0x7fccb80a8a40 2024-01-21T04:33:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.250+0000 7fccb7fff640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccb80a8a40 0x7fccb8136160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcca802c000 tx=0x7fcca8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.250+0000 7fccb4ff9640 10 client.?.objecter ms_handle_connect 0x7fccb80a8a40 2024-01-21T04:33:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.250+0000 7fccb4ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 1 -- 172.21.15.69:0/695675796 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca803c040 con 0x7fccb80a8a40 2024-01-21T04:33:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 10 monclient: dump: 2024-01-21T04:33:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 10 monclient: _finish_auth 0 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 10 monclient: _check_auth_tickets 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 1 -- 172.21.15.69:0/695675796 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca80378f0 con 0x7fccb80a8a40 2024-01-21T04:33:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 1 -- 172.21.15.69:0/695675796 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8037ca0 con 0x7fccb80a8a40 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccbe626640 5 monclient: authenticate success, global_id 4293 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccbe626640 10 monclient: _renew_subs 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccbe626640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccbe626640 1 -- 172.21.15.69:0/695675796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb80b84c0 con 0x7fccb80a8a40 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb5ffb640 4 set_mon_vals no callback set 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccb4ff9640 10 monclient: dump: 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccbe626640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccbe626640 10 monclient: _renew_subs 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccbe626640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccbe626640 1 -- 172.21.15.69:0/695675796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb80b7c60 con 0x7fccb80a8a40 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.253+0000 7fccbe626640 1 librados: init done 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.254+0000 7fccbe626640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.254+0000 7fccbe626640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.254+0000 7fccbe626640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.254+0000 7fccbe626640 1 -- 172.21.15.69:0/695675796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb80b81b0 con 0x7fccb80a8a40 2024-01-21T04:33:49.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.273+0000 7fccb4ff9640 1 -- 172.21.15.69:0/695675796 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fcca804b3b0 con 0x7fccb80a8a40 2024-01-21T04:33:49.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.273+0000 7fccb4ff9640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcca406c8b0 0x7fcca406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.273+0000 7fccb77fe640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcca406c8b0 0x7fcca406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.273+0000 7fccb77fe640 10 monclient: get_auth_request con 0x7fcca406c8b0 auth_method 0 2024-01-21T04:33:49.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.274+0000 7fccb4ff9640 10 client.4293.objecter ms_handle_connect 0x7fcca406c8b0 2024-01-21T04:33:49.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.274+0000 7fccb77fe640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcca406c8b0 0x7fcca406ed70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fccb813a410 tx=0x7fccac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:49.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.276+0000 7fccb4ff9640 1 -- 172.21.15.69:0/695675796 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcca807a800 con 0x7fccb80a8a40 2024-01-21T04:33:49.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.276+0000 7fccb4ff9640 10 client.4293.objecter ms_dispatch 0x7fccb8000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T04:33:49.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.276+0000 7fccb4ff9640 3 client.4293.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T04:33:49.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.276+0000 7fccb4ff9640 3 client.4293.objecter handle_osd_map decoding full epoch 25 2024-01-21T04:33:49.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.276+0000 7fccb4ff9640 20 client.4293.objecter dump_active .. 0 homeless 2024-01-21T04:33:49.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.509+0000 7fccb4ff9640 1 -- 172.21.15.69:0/695675796 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fcca8076b40 con 0x7fccb80a8a40 2024-01-21T04:33:49.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.509+0000 7fccb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:49.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.509+0000 7fccb4ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:49.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.677+0000 7fccbe626640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.677+0000 7fccbe626640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.677+0000 7fccbe626640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:49.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.677+0000 7fccbe626640 1 -- 172.21.15.69:0/695675796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccb804e970 con 0x7fccb80a8a40 2024-01-21T04:33:49.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.709+0000 7fccb4ff9640 1 -- 172.21.15.69:0/695675796 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+14640 (secure 0 0 0) 0x7fcca8047070 con 0x7fccb80a8a40 2024-01-21T04:33:49.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.709+0000 7fccb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T04:33:49.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.709+0000 7fccb4ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:49.711 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:33:49.711 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T04:33:48.398337+0000","last_up_change":"2024-01-21T04:33:38.448164+0000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-21T04:33:29.939892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-21T04:33:45.459402+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6825","nonce":335500148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6827","nonce":335500148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6831","nonce":335500148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6829","nonce":335500148}]},"public_addr":"172.21.15.69:6825/335500148","cluster_addr":"172.21.15.69:6827/335500148","heartbeat_back_addr":"172.21.15.69:6831/335500148","heartbeat_front_addr":"172.21.15.69:6829/335500148","state":["exists","up"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6809","nonce":3868480563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6811","nonce":3868480563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6815","nonce":3868480563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6813","nonce":3868480563}]},"public_addr":"172.21.15.69:6809/3868480563","cluster_addr":"172.21.15.69:6811/3868480563","heartbeat_back_addr":"172.21.15.69:6815/3868480563","heartbeat_front_addr":"172.21.15.69:6813/3868480563","state":["exists","up"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6817","nonce":3954012353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6819","nonce":3954012353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6823","nonce":3954012353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6821","nonce":3954012353}]},"public_addr":"172.21.15.69:6817/3954012353","cluster_addr":"172.21.15.69:6819/3954012353","heartbeat_back_addr":"172.21.15.69:6823/3954012353","heartbeat_front_addr":"172.21.15.69:6821/3954012353","state":["exists","up"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6801","nonce":1086763384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6803","nonce":1086763384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6807","nonce":1086763384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6805","nonce":1086763384}]},"public_addr":"172.21.15.69:6801/1086763384","cluster_addr":"172.21.15.69:6803/1086763384","heartbeat_back_addr":"172.21.15.69:6807/1086763384","heartbeat_front_addr":"172.21.15.69:6805/1086763384","state":["exists","up"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6827","nonce":3897583001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6829","nonce":3897583001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6833","nonce":3897583001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6831","nonce":3897583001}]},"public_addr":"172.21.15.144:6827/3897583001","cluster_addr":"172.21.15.144:6829/3897583001","heartbeat_back_addr":"172.21.15.144:6833/3897583001","heartbeat_front_addr":"172.21.15.144:6831/3897583001","state":["exists","up"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6803","nonce":1782989653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6805","nonce":1782989653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6809","nonce":1782989653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6807","nonce":1782989653}]},"public_addr":"172.21.15.144:6803/1782989653","cluster_addr":"172.21.15.144:6805/1782989653","heartbeat_back_addr":"172.21.15.144:6809/1782989653","heartbeat_front_addr":"172.21.15.144:6807/1782989653","state":["exists","up"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6813","nonce":479183693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6818","nonce":479183693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6825","nonce":479183693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6822","nonce":479183693}]},"public_addr":"172.21.15.144:6813/479183693","cluster_addr":"172.21.15.144:6818/479183693","heartbeat_back_addr":"172.21.15.144:6825/479183693","heartbeat_front_addr":"172.21.15.144:6822/479183693","state":["exists","up"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6811","nonce":14156089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6815","nonce":14156089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6823","nonce":14156089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6819","nonce":14156089}]},"public_addr":"172.21.15.144:6811/14156089","cluster_addr":"172.21.15.144:6815/14156089","heartbeat_back_addr":"172.21.15.144:6823/14156089","heartbeat_front_addr":"172.21.15.144:6819/14156089","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:27.418834+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.672251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.150659+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.544731+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.167108+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:24.415555+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.272052+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.940928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T04:33:49.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 10 librados: watch_flush enter 2024-01-21T04:33:49.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 10 librados: watch_flush exit 2024-01-21T04:33:49.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 20 client.4293.objecter shutdown clearing up homeless session... 2024-01-21T04:33:49.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 10 client.4293.objecter successfully canceled tick 2024-01-21T04:33:49.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 -- 172.21.15.69:0/695675796 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcca406c8b0 msgr2=0x7fcca406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcca406c8b0 0x7fcca406ed70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fccb813a410 tx=0x7fccac009040 comp rx=0 tx=0).stop 2024-01-21T04:33:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 10 monclient: shutdown 2024-01-21T04:33:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 -- 172.21.15.69:0/695675796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccb80a8a40 msgr2=0x7fccb8136160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccb80a8a40 0x7fccb8136160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcca802c000 tx=0x7fcca8004730 comp rx=0 tx=0).stop 2024-01-21T04:33:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 -- 172.21.15.69:0/695675796 shutdown_connections 2024-01-21T04:33:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcca406c8b0 0x7fcca406ed70 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fccb81460a0 0x7fccb81daa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fccb813f8b0 0x7fccb81da500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 --2- 172.21.15.69:0/695675796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccb80a8a40 0x7fccb8136160 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 -- 172.21.15.69:0/695675796 >> 172.21.15.69:0/695675796 conn(0x7fccb8066f50 msgr2=0x7fccb8139ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 -- 172.21.15.69:0/695675796 shutdown_connections 2024-01-21T04:33:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 -- 172.21.15.69:0/695675796 wait complete. 2024-01-21T04:33:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.711+0000 7fccbe626640 1 librados: shutdown 2024-01-21T04:33:49.723 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-21T04:33:49.723 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-21T04:33:49.724 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-21T04:33:49.724 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-21T04:33:49.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-21T04:33:49.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-21T04:33:49.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-21T04:33:49.726 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-21T04:33:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7f8cc111d640 10 monclient: get_monmap_and_config 2024-01-21T04:33:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7f8cc111d640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7f8cc111d640 10 monclient: monmap: 2024-01-21T04:33:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.900 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:49.900616+0000 2024-01-21T04:33:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:49.900616+0000 2024-01-21T04:33:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:49.901 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:49.901 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.897+0000 7fdc5cf6c640 10 monclient: get_monmap_and_config 2024-01-21T04:33:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.897+0000 7fdc5cf6c640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.897+0000 7fdc5cf6c640 10 monclient: monmap: 2024-01-21T04:33:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:49.898394+0000 2024-01-21T04:33:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:49.898394+0000 2024-01-21T04:33:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.902 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.898+0000 7fdc5cf6c640 1 Processor -- start 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.898+0000 7fdc5cf6c640 1 -- start start 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.898+0000 7fdc5cf6c640 10 monclient: init 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-01-21T04:33:49.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc58071ab0 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:49.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdc5806ec10 0x7fdc5806f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 10 monclient(hunting): picked mon.noname-a con 0x7fdc5806ec10 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:49.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc5806f5e0 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:49.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc580ad7d0 con 0x7fdc58071ab0 2024-01-21T04:33:49.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc58072bc0 con 0x7fdc5806f5e0 2024-01-21T04:33:49.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc58071e90 con 0x7fdc5806ec10 2024-01-21T04:33:49.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.899+0000 7fdc5cf6c640 10 monclient(hunting): authenticate will time out at 1388.240797s 2024-01-21T04:33:49.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.900+0000 7f44fad3c640 10 monclient: get_monmap_and_config 2024-01-21T04:33:49.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.900+0000 7f44fad3c640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.900+0000 7f44fad3c640 10 monclient: monmap: 2024-01-21T04:33:49.905 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.905 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.905 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:49.901119+0000 2024-01-21T04:33:49.905 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:49.901119+0000 2024-01-21T04:33:49.905 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.905 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.905 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:49.905 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:49.905 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:49.906 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:49.906 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:49.906 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.901+0000 7f8cc111d640 1 Processor -- start 2024-01-21T04:33:49.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.901+0000 7f44fad3c640 1 Processor -- start 2024-01-21T04:33:49.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.901+0000 7f8cc111d640 1 -- start start 2024-01-21T04:33:49.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.901+0000 7f8cc111d640 10 monclient: init 2024-01-21T04:33:49.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-01-21T04:33:49.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8cbc0aef20 0x7f8cbc1425b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 10 monclient(hunting): picked mon.noname-b con 0x7f8cbc0aef20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8cbc0b2020 0x7f8cbc142af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 10 monclient(hunting): picked mon.noname-c con 0x7f8cbc0b2020 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8cbc0af300 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cbc0b14c0 con 0x7f8cbc0af300 2024-01-21T04:33:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cbc0b02e0 con 0x7f8cbc0b2020 2024-01-21T04:33:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc142090 con 0x7f8cbc0aef20 2024-01-21T04:33:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f8cc111d640 10 monclient(hunting): authenticate will time out at 1388.243491s 2024-01-21T04:33:49.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f44fad3c640 1 -- start start 2024-01-21T04:33:49.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.902+0000 7f44fad3c640 10 monclient: init 2024-01-21T04:33:49.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-01-21T04:33:49.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f44f406f300 0x7f44f406f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 10 monclient(hunting): picked mon.noname-c con 0x7f44f406f300 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:49.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44f413b320 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:49.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f44f413b760 0x7f44f40720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 10 monclient(hunting): picked mon.noname-a con 0x7f44f413b760 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f413ddc0 con 0x7f44f413b320 2024-01-21T04:33:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f4070160 con 0x7f44f413b760 2024-01-21T04:33:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f413ed90 con 0x7f44f406f300 2024-01-21T04:33:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7f44fad3c640 10 monclient(hunting): authenticate will time out at 1388.244326s 2024-01-21T04:33:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.901+0000 7fdc57fff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fdc5806f5e0 0x7fdc5806fa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:53968/0 (socket says 172.21.15.69:53968) 2024-01-21T04:33:49.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7ff334d0f640 10 monclient: get_monmap_and_config 2024-01-21T04:33:49.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7ff334d0f640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.903+0000 7ff334d0f640 10 monclient: monmap: 2024-01-21T04:33:49.912 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.912 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.912 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:49.904677+0000 2024-01-21T04:33:49.912 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:49.904677+0000 2024-01-21T04:33:49.913 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.913 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.913 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:49.913 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:49.913 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:49.913 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:49.913 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:49.913 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.901+0000 7fdc57fff640 1 -- 172.21.15.69:0/1225671724 learned_addr learned my addr 172.21.15.69:0/1225671724 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:49.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f44f9d3a640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f44f406f300 0x7f44f406f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f8cba59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8cbc0b2020 0x7f8cbc142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f8cbad9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8cbc0aef20 0x7f8cbc1425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f44f9539640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f44f413b320 0x7f44f4071990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:53984/0 (socket says 172.21.15.69:53984) 2024-01-21T04:33:49.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f44f9d3a640 10 monclient(hunting): get_auth_request con 0x7f44f406f300 auth_method 0 2024-01-21T04:33:49.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f44f9d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f44f9d3a640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f44f9d3a640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f44f9539640 1 -- 172.21.15.69:0/1765843294 learned_addr learned my addr 172.21.15.69:0/1765843294 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:49.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7ff334d0f640 1 Processor -- start 2024-01-21T04:33:49.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f8cba59c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8cbc0b2020 0x7f8cbc142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59628/0 (socket says 172.21.15.69:59628) 2024-01-21T04:33:49.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f8cba59c640 1 -- 172.21.15.69:0/787080713 learned_addr learned my addr 172.21.15.69:0/787080713 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:49.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f8cba59c640 10 monclient(hunting): get_auth_request con 0x7f8cbc0b2020 auth_method 0 2024-01-21T04:33:49.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f8cba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f8cba59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7f8cba59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.905+0000 7f8cbad9d640 10 monclient(hunting): get_auth_request con 0x7f8cbc0aef20 auth_method 0 2024-01-21T04:33:49.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.905+0000 7f8cbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.905+0000 7f8cbad9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.905+0000 7f8cbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7ff334d0f640 1 -- start start 2024-01-21T04:33:49.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.904+0000 7ff334d0f640 10 monclient: init 2024-01-21T04:33:49.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.905+0000 7f44f9d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.905+0000 7f44f9d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.905+0000 7f44f9d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.905+0000 7ff334d0f640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7ff334d0f640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-01-21T04:33:49.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7ff334d0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3280064d0 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:49.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7ff334d0f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff3280322a0 0x7ff3280cf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7ff334d0f640 10 monclient(hunting): picked mon.noname-c con 0x7ff3280322a0 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7f8cbad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7f8cbad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7f8cbad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7ff334d0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff328033a00 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7ff334d0f640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff328033300 con 0x7ff328033a00 2024-01-21T04:33:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7ff334d0f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3280335e0 con 0x7ff3280064d0 2024-01-21T04:33:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7ff334d0f640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7ff334d0f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3280068b0 con 0x7ff3280322a0 2024-01-21T04:33:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7ff334d0f640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.906+0000 7ff334d0f640 10 monclient(hunting): authenticate will time out at 1388.247709s 2024-01-21T04:33:49.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.907+0000 7f8cba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.907+0000 7f8cba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.907+0000 7f8cba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8c22ad0640 10 monclient: get_monmap_and_config 2024-01-21T04:33:49.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8c22ad0640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8c22ad0640 10 monclient: monmap: 2024-01-21T04:33:49.921 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:49.909316+0000 2024-01-21T04:33:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:49.909316+0000 2024-01-21T04:33:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:49.922 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8cbad9d640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2024-01-21T04:33:49.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8cbad9d640 1 -- 172.21.15.69:0/787080713 >> v2:172.21.15.144:3300/0 conn(0x7f8cbc0b2020 msgr2=0x7f8cbc142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8cbad9d640 1 --2- 172.21.15.69:0/787080713 >> v2:172.21.15.144:3300/0 conn(0x7f8cbc0b2020 0x7f8cbc142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8cbad9d640 1 -- 172.21.15.69:0/787080713 >> v1:172.21.15.69:6789/0 conn(0x7f8cbc0af300 legacy=0x7f8cbc13f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8cbad9d640 10 monclient: _finish_hunting 0 2024-01-21T04:33:49.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8cbad9d640 1 monclient: found mon.noname-b 2024-01-21T04:33:49.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8cbad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:49.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8cbad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:33:49.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f44f9d3a640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2024-01-21T04:33:49.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.908+0000 7f8cbad9d640 1 -- 172.21.15.69:0/787080713 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc0ae6a0 con 0x7f8cbc0aef20 2024-01-21T04:33:49.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f44f9d3a640 1 -- 172.21.15.69:0/1765843294 >> v2:172.21.15.69:3300/0 conn(0x7f44f413b760 msgr2=0x7f44f40720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f44f9d3a640 1 --2- 172.21.15.69:0/1765843294 >> v2:172.21.15.69:3300/0 conn(0x7f44f413b760 0x7f44f40720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f44f9d3a640 1 -- 172.21.15.69:0/1765843294 >> v1:172.21.15.144:6789/0 conn(0x7f44f413b320 legacy=0x7f44f4071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f44f9d3a640 10 monclient: _finish_hunting 0 2024-01-21T04:33:49.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f44f9d3a640 1 monclient: found mon.noname-c 2024-01-21T04:33:49.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f44f9d3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:49.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f44f9d3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:49.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f44f9d3a640 1 -- 172.21.15.69:0/1765843294 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f406fc20 con 0x7f44f406f300 2024-01-21T04:33:49.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f8c22ad0640 1 Processor -- start 2024-01-21T04:33:49.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f8c22ad0640 1 -- start start 2024-01-21T04:33:49.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.909+0000 7f8c22ad0640 10 monclient: init 2024-01-21T04:33:49.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.910+0000 7fdc56ffd640 1 --2- 172.21.15.69:0/1225671724 >> v2:172.21.15.69:3300/0 conn(0x7fdc5806ec10 0x7fdc5806f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.910+0000 7f8c22ad0640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.910+0000 7f8c22ad0640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-01-21T04:33:49.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c1c0a8b20 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:49.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f3b5a105640 10 monclient: get_monmap_and_config 2024-01-21T04:33:49.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f3b5a105640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f3b5a105640 10 monclient: monmap: 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:49.912056+0000 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:49.912056+0000 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:49.929 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0a9d10 0x7f8c1c13f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 10 monclient(hunting): picked mon.noname-b con 0x7f8c1c0a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:49.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8c1c0b2310 0x7f8c1c13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7fdc56ffd640 10 monclient(hunting): get_auth_request con 0x7fdc5806ec10 auth_method 0 2024-01-21T04:33:49.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7fdc56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7fdc56ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7fdc56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 10 monclient(hunting): picked mon.noname-a con 0x7f8c1c0b2310 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:49.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1c1456f0 con 0x7f8c1c0a8b20 2024-01-21T04:33:49.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c1c0a98c0 con 0x7f8c1c0b2310 2024-01-21T04:33:49.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c13fb00 con 0x7f8c1c0a9d10 2024-01-21T04:33:49.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f8c22ad0640 10 monclient(hunting): authenticate will time out at 1388.252966s 2024-01-21T04:33:49.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f3b5a105640 1 Processor -- start 2024-01-21T04:33:49.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f3b5a105640 1 -- start start 2024-01-21T04:33:49.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.911+0000 7f3b5a105640 10 monclient: init 2024-01-21T04:33:49.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f827ad27640 10 monclient: get_monmap_and_config 2024-01-21T04:33:49.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f827ad27640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f827ad27640 10 monclient: monmap: 2024-01-21T04:33:49.933 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.933 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.933 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:49.913109+0000 2024-01-21T04:33:49.933 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:49.913109+0000 2024-01-21T04:33:49.933 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.934 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.934 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:49.934 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:49.934 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:49.934 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:49.934 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:49.934 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-01-21T04:33:49.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b5406a8a0 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:49.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3b540aecb0 0x7f3b54146380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 10 monclient(hunting): picked mon.noname-c con 0x7f3b540aecb0 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:49.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b54143490 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:49.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b540712f0 con 0x7f3b5406a8a0 2024-01-21T04:33:49.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b540715d0 con 0x7f3b54143490 2024-01-21T04:33:49.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b54144030 con 0x7f3b540aecb0 2024-01-21T04:33:49.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f827ad27640 1 Processor -- start 2024-01-21T04:33:49.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f8cbad9d640 1 --2- 172.21.15.69:0/787080713 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8cbc0aef20 0x7f8cbc1425b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009d90 tx=0x7f8cb402be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b082ce59756a349e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:49.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.912+0000 7f3b5a105640 10 monclient(hunting): authenticate will time out at 1388.253798s 2024-01-21T04:33:49.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f827ad27640 1 -- start start 2024-01-21T04:33:49.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f827ad27640 10 monclient: init 2024-01-21T04:33:49.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f8c1bfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0a9d10 0x7f8c1c13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f827ad27640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f827ad27640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-01-21T04:33:49.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f8c1bfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0a9d10 0x7f8c1c13f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46806/0 (socket says 172.21.15.69:46806) 2024-01-21T04:33:49.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f8c1bfff640 1 -- 172.21.15.69:0/2485315737 learned_addr learned my addr 172.21.15.69:0/2485315737 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:49.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f8c1bfff640 10 monclient(hunting): get_auth_request con 0x7f8c1c0a9d10 auth_method 0 2024-01-21T04:33:49.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f8c1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f8c1bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f8c1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f827ad27640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f826c035070 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:49.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f827ad27640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 0x7f826c0c3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.913+0000 7f827ad27640 10 monclient(hunting): picked mon.noname-b con 0x7f826c0c2e20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:49.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.914+0000 7f827ad27640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f826c0c8fb0 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:49.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.914+0000 7f827ad27640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c02d610 con 0x7f826c0c8fb0 2024-01-21T04:33:49.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.914+0000 7f827ad27640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c02ea30 con 0x7f826c035070 2024-01-21T04:33:49.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.914+0000 7f827ad27640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.914+0000 7f827ad27640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c035450 con 0x7f826c0c2e20 2024-01-21T04:33:49.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.914+0000 7f827ad27640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.914+0000 7f827ad27640 10 monclient(hunting): authenticate will time out at 1388.255345s 2024-01-21T04:33:49.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.914+0000 7f8c1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.914+0000 7f8c1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.914+0000 7f8c1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.914+0000 7ff32dd9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff3280322a0 0x7ff3280cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.915+0000 7ff32dd9b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff3280322a0 0x7ff3280cf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59652/0 (socket says 172.21.15.69:59652) 2024-01-21T04:33:49.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.915+0000 7ff32dd9b640 1 -- 172.21.15.69:0/878274394 learned_addr learned my addr 172.21.15.69:0/878274394 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:49.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.915+0000 7ff32dd9b640 10 monclient(hunting): get_auth_request con 0x7ff3280322a0 auth_method 0 2024-01-21T04:33:49.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.915+0000 7ff32dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.915+0000 7ff32dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.915+0000 7ff32dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.915+0000 7f3b52ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3b540aecb0 0x7f3b54146380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.916+0000 7ff32dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.916+0000 7ff32dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.916+0000 7ff32dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.916+0000 7f3b52ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f3b540aecb0 0x7f3b54146380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59668/0 (socket says 172.21.15.69:59668) 2024-01-21T04:33:49.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.916+0000 7f3b52ffd640 1 -- 172.21.15.69:0/1042719104 learned_addr learned my addr 172.21.15.69:0/1042719104 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:49.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.916+0000 7f3b52ffd640 10 monclient(hunting): get_auth_request con 0x7f3b540aecb0 auth_method 0 2024-01-21T04:33:49.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.916+0000 7f3b52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.916+0000 7f3b52ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.916+0000 7f3b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.917+0000 7ff32dd9b640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2024-01-21T04:33:49.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.917+0000 7ff32dd9b640 1 -- 172.21.15.69:0/878274394 >> v1:172.21.15.144:6789/0 conn(0x7ff3280064d0 legacy=0x7ff328031b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.917+0000 7ff32dd9b640 1 -- 172.21.15.69:0/878274394 >> v1:172.21.15.69:6789/0 conn(0x7ff328033a00 legacy=0x7ff3280cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.917+0000 7ff32dd9b640 10 monclient: _finish_hunting 0 2024-01-21T04:33:49.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.917+0000 7ff32dd9b640 1 monclient: found mon.noname-c 2024-01-21T04:33:49.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.917+0000 7ff32dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:49.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.917+0000 7ff32dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:49.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.917+0000 7ff32dd9b640 1 -- 172.21.15.69:0/878274394 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff328030fe0 con 0x7ff3280322a0 2024-01-21T04:33:49.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f8c1bfff640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-01-21T04:33:49.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f8c1bfff640 1 -- 172.21.15.69:0/2485315737 >> v2:172.21.15.69:3300/0 conn(0x7f8c1c0b2310 msgr2=0x7f8c1c13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f8c1bfff640 1 --2- 172.21.15.69:0/2485315737 >> v2:172.21.15.69:3300/0 conn(0x7f8c1c0b2310 0x7f8c1c13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f8c1bfff640 1 -- 172.21.15.69:0/2485315737 >> v1:172.21.15.69:6789/0 conn(0x7f8c1c0a8b20 legacy=0x7f8c1c145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f8c1bfff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:49.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f8c1bfff640 1 monclient: found mon.noname-b 2024-01-21T04:33:49.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f8c1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:49.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f8c1bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:33:49.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.918+0000 7f8c1bfff640 1 -- 172.21.15.69:0/2485315737 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c146290 con 0x7f8c1c0a9d10 2024-01-21T04:33:49.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.920+0000 7fdc56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.920+0000 7fdc56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.920+0000 7fdc56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.920+0000 7f022b3b6640 10 monclient: get_monmap_and_config 2024-01-21T04:33:49.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.920+0000 7f022b3b6640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.920+0000 7f022b3b6640 10 monclient: monmap: 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:49.921685+0000 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:49.921685+0000 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f022b3b6640 1 Processor -- start 2024-01-21T04:33:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f022b3b6640 1 -- start start 2024-01-21T04:33:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f022b3b6640 10 monclient: init 2024-01-21T04:33:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f3b52ffd640 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2024-01-21T04:33:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f8279524640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 0x7f826c0c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f3b52ffd640 1 -- 172.21.15.69:0/1042719104 >> v1:172.21.15.144:6789/0 conn(0x7f3b54143490 legacy=0x7f3b54143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f3b52ffd640 1 -- 172.21.15.69:0/1042719104 >> v1:172.21.15.69:6789/0 conn(0x7f3b5406a8a0 legacy=0x7f3b54072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f3b52ffd640 10 monclient: _finish_hunting 0 2024-01-21T04:33:49.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f3b52ffd640 1 monclient: found mon.noname-c 2024-01-21T04:33:49.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f3b52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:49.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f8279524640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 0x7f826c0c3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46810/0 (socket says 172.21.15.69:46810) 2024-01-21T04:33:49.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f8279524640 1 -- 172.21.15.69:0/863641787 learned_addr learned my addr 172.21.15.69:0/863641787 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:49.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f8279524640 10 monclient(hunting): get_auth_request con 0x7f826c0c2e20 auth_method 0 2024-01-21T04:33:49.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f8279524640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f8279524640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f8279524640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f3b52ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:49.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.921+0000 7f3b52ffd640 1 -- 172.21.15.69:0/1042719104 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b54071d20 con 0x7f3b540aecb0 2024-01-21T04:33:49.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-01-21T04:33:49.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f02240a8b20 0x7f0224145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 10 monclient(hunting): picked mon.noname-a con 0x7f02240a8b20 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:49.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02240a9d10 0x7f022413f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 10 monclient(hunting): picked mon.noname-b con 0x7f02240a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:49.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02240b2310 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:49.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02241456f0 con 0x7f02240b2310 2024-01-21T04:33:49.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02240a98c0 con 0x7f02240a8b20 2024-01-21T04:33:49.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f8279524640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f8279524640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f8279524640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02241460c0 con 0x7f02240a9d10 2024-01-21T04:33:49.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7f022b3b6640 10 monclient(hunting): authenticate will time out at 1388.264082s 2024-01-21T04:33:49.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7fdc56ffd640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-01-21T04:33:49.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.922+0000 7fdc56ffd640 1 -- 172.21.15.69:0/1225671724 >> v1:172.21.15.144:6789/0 conn(0x7fdc5806f5e0 legacy=0x7fdc5806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7fdc56ffd640 1 -- 172.21.15.69:0/1225671724 >> v1:172.21.15.69:6789/0 conn(0x7fdc58071ab0 legacy=0x7fdc580ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7fdc56ffd640 10 monclient: _finish_hunting 0 2024-01-21T04:33:49.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7fdc56ffd640 1 monclient: found mon.noname-a 2024-01-21T04:33:49.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7fdc56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:49.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7fdc56ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:49.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7fdc56ffd640 1 -- 172.21.15.69:0/1225671724 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc58070a80 con 0x7fdc5806ec10 2024-01-21T04:33:49.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7ff32dd9b640 1 --2- 172.21.15.69:0/878274394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3280322a0 0x7ff3280cf610 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff328033eb0 tx=0x7ff32402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fd3b3e1083fcc94 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:49.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 1 -- 172.21.15.69:0/787080713 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb402cbd0 con 0x7f8cbc0aef20 2024-01-21T04:33:49.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:49.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 10 monclient: dump: 2024-01-21T04:33:49.959 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.959 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.959 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 10 monclient: _finish_auth 0 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 10 monclient: _check_auth_tickets 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 1 -- 172.21.15.69:0/787080713 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb402cd30 con 0x7f8cbc0aef20 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:49.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cc111d640 5 monclient: authenticate success, global_id 4265 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 1 -- 172.21.15.69:0/787080713 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb402cea0 con 0x7f8cbc0aef20 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cc111d640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cc111d640 4 set_mon_vals no callback set 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cb9d9b640 10 monclient: dump: 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.961 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cc111d640 10 monclient: shutdown 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cc111d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cc111d640 1 -- 172.21.15.69:0/787080713 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8cbc0aef20 msgr2=0x7f8cbc1425b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cc111d640 1 --2- 172.21.15.69:0/787080713 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8cbc0aef20 0x7f8cbc1425b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009d90 tx=0x7f8cb402be20 comp rx=0 tx=0).stop 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cc111d640 1 -- 172.21.15.69:0/787080713 shutdown_connections 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cc111d640 1 --2- 172.21.15.69:0/787080713 >> v2:172.21.15.144:3300/0 conn(0x7f8cbc0b2020 0x7f8cbc142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cc111d640 1 --2- 172.21.15.69:0/787080713 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8cbc0aef20 0x7f8cbc1425b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.923+0000 7f8cc111d640 1 -- 172.21.15.69:0/787080713 >> 172.21.15.69:0/787080713 conn(0x7f8cbc066cd0 msgr2=0x7f8cbc0b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.924+0000 7f8cc111d640 1 -- 172.21.15.69:0/787080713 shutdown_connections 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.924+0000 7f8cc111d640 1 -- 172.21.15.69:0/787080713 wait complete. 2024-01-21T04:33:49.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.924+0000 7f8cc111d640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.924+0000 7f8cc111d640 10 monclient: monmap: 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8cc111d640 1 librados: starting msgr at 2024-01-21T04:33:49.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8cc111d640 1 librados: starting objecter 2024-01-21T04:33:49.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8279524640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-01-21T04:33:49.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8279524640 1 -- 172.21.15.69:0/863641787 >> v1:172.21.15.144:6789/0 conn(0x7f826c035070 legacy=0x7f826c02cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8cc111d640 1 Processor -- start 2024-01-21T04:33:49.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8cc111d640 1 -- start start 2024-01-21T04:33:49.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8cc111d640 1 librados: setting wanted keys 2024-01-21T04:33:49.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8cc111d640 1 librados: calling monclient init 2024-01-21T04:33:49.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8cc111d640 10 monclient: init 2024-01-21T04:33:49.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8279524640 1 -- 172.21.15.69:0/863641787 >> v1:172.21.15.69:6789/0 conn(0x7f826c0c8fb0 legacy=0x7f826c0c9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8279524640 10 monclient: _finish_hunting 0 2024-01-21T04:33:49.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8279524640 1 monclient: found mon.noname-b 2024-01-21T04:33:49.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8279524640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:49.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8279524640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:33:49.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.925+0000 7f8279524640 1 -- 172.21.15.69:0/863641787 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c035c80 con 0x7f826c0c2e20 2024-01-21T04:33:49.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8c1bfff640 1 --2- 172.21.15.69:0/2485315737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0a9d10 0x7f8c1c13f080 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8c04009870 tx=0x7f8c0402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdcdd94ed3b7a16e server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:49.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T04:33:49.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8cbc0aef20 0x7f8cbc145ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 10 monclient(hunting): picked mon.noname-c con 0x7f8cbc0aef20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:49.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8cbc0af300 0x7f8cbc143260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 10 monclient(hunting): picked mon.noname-a con 0x7f8cbc0af300 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:49.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cbc0b2020 0x7f8cbc1d81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 10 monclient(hunting): picked mon.noname-b con 0x7f8cbc0b2020 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:49.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc1d7d60 con 0x7f8cbc0af300 2024-01-21T04:33:49.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc1437a0 con 0x7f8cbc0b2020 2024-01-21T04:33:49.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc145450 con 0x7f8cbc0aef20 2024-01-21T04:33:49.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.926+0000 7f8cc111d640 10 monclient(hunting): authenticate will time out at 1388.267559s 2024-01-21T04:33:49.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.928+0000 7f44f9d3a640 1 --2- 172.21.15.69:0/1765843294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44f406f300 0x7f44f406f6e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f44f406e870 tx=0x7f44e402c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=438fde1ca64015a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:49.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.929+0000 7f8279524640 1 --2- 172.21.15.69:0/863641787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 0x7f826c0c3260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8264009880 tx=0x7f826402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be5e49cb470747c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:49.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 1 -- 172.21.15.69:0/878274394 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff324031830 con 0x7ff3280322a0 2024-01-21T04:33:49.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:49.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 10 monclient: dump: 2024-01-21T04:33:49.970 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 10 monclient: _finish_auth 0 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 10 monclient: _check_auth_tickets 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:49.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 1 -- 172.21.15.69:0/878274394 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff324031990 con 0x7ff3280322a0 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 1 -- 172.21.15.69:0/878274394 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff324031d20 con 0x7ff3280322a0 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.930+0000 7ff32d59a640 10 monclient: dump: 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.972 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.931+0000 7ff334d0f640 5 monclient: authenticate success, global_id 4312 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.931+0000 7ff334d0f640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.931+0000 7ff334d0f640 4 set_mon_vals no callback set 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.931+0000 7ff334d0f640 10 monclient: shutdown 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.931+0000 7ff334d0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.931+0000 7ff334d0f640 1 -- 172.21.15.69:0/878274394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3280322a0 msgr2=0x7ff3280cf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.931+0000 7ff334d0f640 1 --2- 172.21.15.69:0/878274394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3280322a0 0x7ff3280cf610 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff328033eb0 tx=0x7ff32402b8d0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.931+0000 7ff334d0f640 1 -- 172.21.15.69:0/878274394 shutdown_connections 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.931+0000 7ff334d0f640 1 --2- 172.21.15.69:0/878274394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff3280322a0 0x7ff3280cf610 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.931+0000 7ff334d0f640 1 -- 172.21.15.69:0/878274394 >> 172.21.15.69:0/878274394 conn(0x7ff328006af0 msgr2=0x7ff328006f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.932+0000 7ff334d0f640 1 -- 172.21.15.69:0/878274394 shutdown_connections 2024-01-21T04:33:49.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.932+0000 7ff334d0f640 1 -- 172.21.15.69:0/878274394 wait complete. 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.933+0000 7ff334d0f640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.933+0000 7ff334d0f640 10 monclient: monmap: 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.933+0000 7ff334d0f640 1 librados: starting msgr at 2024-01-21T04:33:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.933+0000 7ff334d0f640 1 librados: starting objecter 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.933+0000 7ff334d0f640 1 Processor -- start 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.933+0000 7ff334d0f640 1 -- start start 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.933+0000 7ff334d0f640 1 librados: setting wanted keys 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.933+0000 7ff334d0f640 1 librados: calling monclient init 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.933+0000 7ff334d0f640 10 monclient: init 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3280064d0 0x7ff3280c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 10 monclient(hunting): picked mon.noname-a con 0x7ff3280064d0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3280322a0 0x7ff3280c08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 10 monclient(hunting): picked mon.noname-c con 0x7ff3280322a0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff328033a00 0x7ff3280c0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 10 monclient(hunting): picked mon.noname-b con 0x7ff328033a00 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32802b640 con 0x7ff3280064d0 2024-01-21T04:33:49.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32802a3a0 con 0x7ff328033a00 2024-01-21T04:33:49.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32802b8b0 con 0x7ff3280322a0 2024-01-21T04:33:49.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.934+0000 7ff334d0f640 10 monclient(hunting): authenticate will time out at 1388.276054s 2024-01-21T04:33:49.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.936+0000 7f0228951640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02240a9d10 0x7f022413f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 1 -- 172.21.15.69:0/2485315737 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04031640 con 0x7f8c1c0a9d10 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 10 monclient: dump: 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.977 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 1 -- 172.21.15.69:0/2485315737 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c040317a0 con 0x7f8c1c0a9d10 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c22ad0640 5 monclient: authenticate success, global_id 4268 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c22ad0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c22ad0640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c22ad0640 4 set_mon_vals no callback set 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 1 -- 172.21.15.69:0/2485315737 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04031b30 con 0x7f8c1c0a9d10 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f8c1b7fe640 10 monclient: dump: 2024-01-21T04:33:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.979 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f0228951640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02240a9d10 0x7f022413f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46812/0 (socket says 172.21.15.69:46812) 2024-01-21T04:33:49.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f0228951640 1 -- 172.21.15.69:0/3219509575 learned_addr learned my addr 172.21.15.69:0/3219509575 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:49.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f0228951640 10 monclient(hunting): get_auth_request con 0x7f02240a9d10 auth_method 0 2024-01-21T04:33:49.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f0228951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f0228951640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.937+0000 7f0228951640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f8c22ad0640 10 monclient: shutdown 2024-01-21T04:33:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f8c22ad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f8c22ad0640 1 -- 172.21.15.69:0/2485315737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0a9d10 msgr2=0x7f8c1c13f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f8c22ad0640 1 --2- 172.21.15.69:0/2485315737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0a9d10 0x7f8c1c13f080 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8c04009870 tx=0x7f8c0402b8f0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f8c22ad0640 1 -- 172.21.15.69:0/2485315737 shutdown_connections 2024-01-21T04:33:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f8c22ad0640 1 --2- 172.21.15.69:0/2485315737 >> v2:172.21.15.69:3300/0 conn(0x7f8c1c0b2310 0x7f8c1c13f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f8c22ad0640 1 --2- 172.21.15.69:0/2485315737 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0a9d10 0x7f8c1c13f080 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f8c22ad0640 1 -- 172.21.15.69:0/2485315737 >> 172.21.15.69:0/2485315737 conn(0x7f8c1c066fb0 msgr2=0x7f8c1c0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f8c22ad0640 1 -- 172.21.15.69:0/2485315737 shutdown_connections 2024-01-21T04:33:49.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f8c22ad0640 1 -- 172.21.15.69:0/2485315737 wait complete. 2024-01-21T04:33:49.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f0228951640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f0228951640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f0228951640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.938+0000 7f8cbb59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cbc0b2020 0x7f8cbc1d81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8c22ad0640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8c22ad0640 10 monclient: monmap: 2024-01-21T04:33:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:49.984 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8c22ad0640 1 librados: starting msgr at 2024-01-21T04:33:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8c22ad0640 1 librados: starting objecter 2024-01-21T04:33:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8c22ad0640 1 Processor -- start 2024-01-21T04:33:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8cbb59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cbc0b2020 0x7f8cbc1d81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59670/0 (socket says 172.21.15.69:59670) 2024-01-21T04:33:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8cbb59e640 1 -- 172.21.15.69:0/3774713728 learned_addr learned my addr 172.21.15.69:0/3774713728 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8cbb59e640 10 monclient(hunting): get_auth_request con 0x7f8cbc0b2020 auth_method 0 2024-01-21T04:33:49.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8cbb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8cbb59e640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8cbb59e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8c22ad0640 1 -- start start 2024-01-21T04:33:49.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8c22ad0640 1 librados: setting wanted keys 2024-01-21T04:33:49.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8c22ad0640 1 librados: calling monclient init 2024-01-21T04:33:49.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.939+0000 7f8c22ad0640 10 monclient: init 2024-01-21T04:33:49.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f3b52ffd640 1 --2- 172.21.15.69:0/1042719104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b540aecb0 0x7f3b54146380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b480098a0 tx=0x7f3b4802b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fdf79e751fb783c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:49.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f0228951640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-01-21T04:33:49.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f0228951640 1 -- 172.21.15.69:0/3219509575 >> v2:172.21.15.69:3300/0 conn(0x7f02240a8b20 msgr2=0x7f0224145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f0228951640 1 --2- 172.21.15.69:0/3219509575 >> v2:172.21.15.69:3300/0 conn(0x7f02240a8b20 0x7f0224145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f0228951640 1 -- 172.21.15.69:0/3219509575 >> v1:172.21.15.144:6789/0 conn(0x7f02240b2310 legacy=0x7f022413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:49.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f0228951640 10 monclient: _finish_hunting 0 2024-01-21T04:33:49.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f0228951640 1 monclient: found mon.noname-b 2024-01-21T04:33:49.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f0228951640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:49.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f0228951640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:33:49.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f0228951640 1 -- 172.21.15.69:0/3219509575 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0224146230 con 0x7f02240a9d10 2024-01-21T04:33:49.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7ff32dd9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3280322a0 0x7ff3280c08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:49.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44f8d38640 1 -- 172.21.15.69:0/1765843294 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4008df0 con 0x7f44f406f300 2024-01-21T04:33:49.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44f8d38640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44f8d38640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44f8d38640 10 monclient: dump: 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44f8d38640 10 monclient: _finish_auth 0 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44f8d38640 10 monclient: _check_auth_tickets 2024-01-21T04:33:49.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44f8d38640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:49.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44fad3c640 5 monclient: authenticate success, global_id 4306 2024-01-21T04:33:49.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44fad3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:49.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44f8d38640 1 -- 172.21.15.69:0/1765843294 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e400a0d0 con 0x7f44f406f300 2024-01-21T04:33:49.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44f8d38640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:49.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44fad3c640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:49.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f44fad3c640 4 set_mon_vals no callback set 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c1c0a8b20 0x7f8c1c142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 10 monclient(hunting): picked mon.noname-b con 0x7f8c1c0a8b20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0a9d10 0x7f8c1c137e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 10 monclient(hunting): picked mon.noname-c con 0x7f8c1c0a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c1c0b2310 0x7f8c1c140100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 10 monclient(hunting): picked mon.noname-a con 0x7f8c1c0b2310 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c139860 con 0x7f8c1c0b2310 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c1385c0 con 0x7f8c1c0a8b20 2024-01-21T04:33:49.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c139a40 con 0x7f8c1c0a9d10 2024-01-21T04:33:49.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:49.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.940+0000 7f8c22ad0640 10 monclient(hunting): authenticate will time out at 1388.281758s 2024-01-21T04:33:49.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f8cbb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f8cbb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f8cbb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44f8d38640 1 -- 172.21.15.69:0/1765843294 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e400a460 con 0x7f44f406f300 2024-01-21T04:33:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44f8d38640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44f8d38640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44f8d38640 10 monclient: dump: 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44fad3c640 10 monclient: shutdown 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44fad3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44fad3c640 1 -- 172.21.15.69:0/1765843294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44f406f300 msgr2=0x7f44f406f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44fad3c640 1 --2- 172.21.15.69:0/1765843294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44f406f300 0x7f44f406f6e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f44f406e870 tx=0x7f44e402c9f0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44fad3c640 1 -- 172.21.15.69:0/1765843294 shutdown_connections 2024-01-21T04:33:49.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44fad3c640 1 --2- 172.21.15.69:0/1765843294 >> v2:172.21.15.69:3300/0 conn(0x7f44f413b760 0x7f44f40720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44fad3c640 1 --2- 172.21.15.69:0/1765843294 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44f406f300 0x7f44f406f6e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:49.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44fad3c640 1 -- 172.21.15.69:0/1765843294 >> 172.21.15.69:0/1765843294 conn(0x7f44f40b0040 msgr2=0x7f44f40b0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:49.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44fad3c640 1 -- 172.21.15.69:0/1765843294 shutdown_connections 2024-01-21T04:33:49.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7f44fad3c640 1 -- 172.21.15.69:0/1765843294 wait complete. 2024-01-21T04:33:49.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7ff32dd9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3280322a0 0x7ff3280c08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46816/0 (socket says 172.21.15.69:46816) 2024-01-21T04:33:49.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7ff32dd9b640 1 -- 172.21.15.69:0/4075847385 learned_addr learned my addr 172.21.15.69:0/4075847385 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:49.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7ff32dd9b640 10 monclient(hunting): get_auth_request con 0x7ff3280322a0 auth_method 0 2024-01-21T04:33:49.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7ff32dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:49.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7ff32dd9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:49.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.941+0000 7ff32dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:49.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.942+0000 7f44fad3c640 10 monclient: build_initial_monmap 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.942+0000 7f44fad3c640 10 monclient: monmap: 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.943+0000 7f44fad3c640 1 librados: starting msgr at 2024-01-21T04:33:49.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.943+0000 7f44fad3c640 1 librados: starting objecter 2024-01-21T04:33:49.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.943+0000 7f44fad3c640 1 Processor -- start 2024-01-21T04:33:49.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.943+0000 7f44fad3c640 1 -- start start 2024-01-21T04:33:49.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.943+0000 7f44fad3c640 1 librados: setting wanted keys 2024-01-21T04:33:49.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.943+0000 7f44fad3c640 1 librados: calling monclient init 2024-01-21T04:33:49.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.943+0000 7f44fad3c640 10 monclient: init 2024-01-21T04:33:49.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.943+0000 7ff32dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:49.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.943+0000 7ff32dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:49.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.943+0000 7ff32dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:49.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:49.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:33:49.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44f406f300 0x7f44f4141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 10 monclient(hunting): picked mon.noname-a con 0x7f44f406f300 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:49.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44f413b320 0x7f44f40ac510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 10 monclient(hunting): picked mon.noname-c con 0x7f44f413b320 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:49.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44f413b760 0x7f44f4136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:49.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 10 monclient(hunting): picked mon.noname-b con 0x7f44f413b760 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:49.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:49.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f40ad390 con 0x7f44f406f300 2024-01-21T04:33:50.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f40ace70 con 0x7f44f413b760 2024-01-21T04:33:50.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f40abd10 con 0x7f44f413b320 2024-01-21T04:33:50.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f44fad3c640 10 monclient(hunting): authenticate will time out at 1388.285476s 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f8278d23640 1 -- 172.21.15.69:0/863641787 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264004230 con 0x7f826c0c2e20 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f8278d23640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f8278d23640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.944+0000 7f8278d23640 10 monclient: dump: 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.001 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.002 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8278d23640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8278d23640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8278d23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8278d23640 1 -- 172.21.15.69:0/863641787 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8264004390 con 0x7f826c0c2e20 2024-01-21T04:33:50.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f827ad27640 5 monclient: authenticate success, global_id 4271 2024-01-21T04:33:50.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f827ad27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:50.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8278d23640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f0228951640 1 --2- 172.21.15.69:0/3219509575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02240a9d10 0x7f022413f080 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0214009870 tx=0x7f021402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71b95dc23a915e9f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8278d23640 1 -- 172.21.15.69:0/863641787 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264031830 con 0x7f826c0c2e20 2024-01-21T04:33:50.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8278d23640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8278d23640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:50.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8278d23640 10 monclient: dump: 2024-01-21T04:33:50.003 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f827ad27640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f827ad27640 4 set_mon_vals no callback set 2024-01-21T04:33:50.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8cbb59e640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-01-21T04:33:50.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8cbb59e640 1 -- 172.21.15.69:0/3774713728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8cbc0aef20 msgr2=0x7f8cbc145ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8cbb59e640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8cbc0aef20 0x7f8cbc145ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8cbb59e640 1 -- 172.21.15.69:0/3774713728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8cbc0af300 msgr2=0x7f8cbc143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8cbb59e640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8cbc0af300 0x7f8cbc143260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8cbb59e640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8cbb59e640 1 monclient: found mon.noname-b 2024-01-21T04:33:50.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8cbb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8cbb59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f8cbb59e640 1 -- 172.21.15.69:0/3774713728 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb4009a70 con 0x7f8cbc0b2020 2024-01-21T04:33:50.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f827ad27640 10 monclient: shutdown 2024-01-21T04:33:50.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f827ad27640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f827ad27640 1 -- 172.21.15.69:0/863641787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 msgr2=0x7f826c0c3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.945+0000 7f827ad27640 1 --2- 172.21.15.69:0/863641787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 0x7f826c0c3260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8264009880 tx=0x7f826402b8d0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.946+0000 7f827ad27640 1 -- 172.21.15.69:0/863641787 shutdown_connections 2024-01-21T04:33:50.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.946+0000 7f827ad27640 1 --2- 172.21.15.69:0/863641787 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 0x7f826c0c3260 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.946+0000 7f827ad27640 1 -- 172.21.15.69:0/863641787 >> 172.21.15.69:0/863641787 conn(0x7f826c006510 msgr2=0x7f826c006920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.946+0000 7f827ad27640 1 -- 172.21.15.69:0/863641787 shutdown_connections 2024-01-21T04:33:50.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.946+0000 7f827ad27640 1 -- 172.21.15.69:0/863641787 wait complete. 2024-01-21T04:33:50.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.946+0000 7ff32ed9d640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff328033a00 0x7ff3280c0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.946+0000 7f827ad27640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.946+0000 7f827ad27640 10 monclient: monmap: 2024-01-21T04:33:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.947+0000 7f827ad27640 1 librados: starting msgr at 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.947+0000 7f827ad27640 1 librados: starting objecter 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.947+0000 7f827ad27640 1 Processor -- start 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.947+0000 7f827ad27640 1 -- start start 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.947+0000 7f827ad27640 1 librados: setting wanted keys 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.947+0000 7f827ad27640 1 librados: calling monclient init 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.947+0000 7f827ad27640 10 monclient: init 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.947+0000 7f827ad27640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.947+0000 7f827ad27640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.947+0000 7f827ad27640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826c035070 0x7f826c0c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.947+0000 7f827ad27640 10 monclient(hunting): picked mon.noname-a con 0x7f826c035070 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 0x7f826c0c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 10 monclient(hunting): picked mon.noname-c con 0x7f826c0c2e20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f826c0c8fb0 0x7f826c0b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 10 monclient(hunting): picked mon.noname-b con 0x7f826c0c8fb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0c47e0 con 0x7f826c035070 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0c4ac0 con 0x7f826c0c8fb0 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0c4da0 con 0x7f826c0c2e20 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7f827ad27640 10 monclient(hunting): authenticate will time out at 1388.289347s 2024-01-21T04:33:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7ff32dd9b640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-01-21T04:33:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7ff32dd9b640 1 -- 172.21.15.69:0/4075847385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff328033a00 msgr2=0x7ff3280c0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7ff32dd9b640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff328033a00 0x7ff3280c0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7ff32dd9b640 1 -- 172.21.15.69:0/4075847385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3280064d0 msgr2=0x7ff3280c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7ff32dd9b640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff3280064d0 0x7ff3280c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7ff32dd9b640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7ff32dd9b640 1 monclient: found mon.noname-c 2024-01-21T04:33:50.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.948+0000 7ff32dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.949+0000 7ff32dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.949+0000 7ff32dd9b640 1 -- 172.21.15.69:0/4075847385 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240094d0 con 0x7ff3280322a0 2024-01-21T04:33:50.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 1 -- 172.21.15.69:0/1042719104 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b48031740 con 0x7f3b540aecb0 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 10 monclient: dump: 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 1 -- 172.21.15.69:0/1042719104 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b480318a0 con 0x7f3b540aecb0 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 1 -- 172.21.15.69:0/1042719104 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b48031c30 con 0x7f3b540aecb0 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b527fc640 10 monclient: dump: 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.015 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.016 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.016 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.016 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.016 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b5a105640 5 monclient: authenticate success, global_id 4315 2024-01-21T04:33:50.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b5a105640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:50.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.950+0000 7f3b5a105640 4 set_mon_vals no callback set 2024-01-21T04:33:50.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f44fa53b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44f413b760 0x7f44f4136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f3b5a105640 10 monclient: shutdown 2024-01-21T04:33:50.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f3b5a105640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f3b5a105640 1 -- 172.21.15.69:0/1042719104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b540aecb0 msgr2=0x7f3b54146380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f3b5a105640 1 --2- 172.21.15.69:0/1042719104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b540aecb0 0x7f3b54146380 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b480098a0 tx=0x7f3b4802b7e0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f3b5a105640 1 -- 172.21.15.69:0/1042719104 shutdown_connections 2024-01-21T04:33:50.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f3b5a105640 1 --2- 172.21.15.69:0/1042719104 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b540aecb0 0x7f3b54146380 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f3b5a105640 1 -- 172.21.15.69:0/1042719104 >> 172.21.15.69:0/1042719104 conn(0x7f3b540b4c60 msgr2=0x7f3b540b5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f3b5a105640 1 -- 172.21.15.69:0/1042719104 shutdown_connections 2024-01-21T04:33:50.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f3b5a105640 1 -- 172.21.15.69:0/1042719104 wait complete. 2024-01-21T04:33:50.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f44fa53b640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44f413b760 0x7f44f4136850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59696/0 (socket says 172.21.15.69:59696) 2024-01-21T04:33:50.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f44fa53b640 1 -- 172.21.15.69:0/1822962835 learned_addr learned my addr 172.21.15.69:0/1822962835 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f44fa53b640 10 monclient(hunting): get_auth_request con 0x7f44f413b760 auth_method 0 2024-01-21T04:33:50.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f44fa53b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f44fa53b640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.951+0000 7f44fa53b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f3b5a105640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f3b5a105640 10 monclient: monmap: 2024-01-21T04:33:50.019 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.019 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:50.019 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:50.020 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:50.020 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.020 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.020 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:50.020 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:50.020 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:50.020 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f44fa53b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f44fa53b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f44fa53b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f3b5a105640 1 librados: starting msgr at 2024-01-21T04:33:50.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f3b5a105640 1 librados: starting objecter 2024-01-21T04:33:50.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f3b5a105640 1 Processor -- start 2024-01-21T04:33:50.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f3b5a105640 1 -- start start 2024-01-21T04:33:50.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f3b5a105640 1 librados: setting wanted keys 2024-01-21T04:33:50.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f3b5a105640 1 librados: calling monclient init 2024-01-21T04:33:50.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.952+0000 7f3b5a105640 10 monclient: init 2024-01-21T04:33:50.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f8c2086c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c1c0a8b20 0x7f8c1c142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:33:50.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b5406a8a0 0x7f3b5413ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 10 monclient(hunting): picked mon.noname-a con 0x7f3b5406a8a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b540aecb0 0x7f3b54141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 10 monclient(hunting): picked mon.noname-b con 0x7f3b540aecb0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b54143490 0x7f3b5413f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 10 monclient(hunting): picked mon.noname-c con 0x7f3b54143490 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54141e80 con 0x7f3b5406a8a0 2024-01-21T04:33:50.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b541422a0 con 0x7f3b540aecb0 2024-01-21T04:33:50.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b54140d50 con 0x7f3b54143490 2024-01-21T04:33:50.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.953+0000 7f3b5a105640 10 monclient(hunting): authenticate will time out at 1388.295123s 2024-01-21T04:33:50.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.955+0000 7f44fa53b640 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2024-01-21T04:33:50.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.955+0000 7f44fa53b640 1 -- 172.21.15.69:0/1822962835 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44f413b320 msgr2=0x7f44f40ac510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.955+0000 7f44fa53b640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f44f413b320 0x7f44f40ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.955+0000 7f44fa53b640 1 -- 172.21.15.69:0/1822962835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44f406f300 msgr2=0x7f44f4141330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.955+0000 7f44fa53b640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44f406f300 0x7f44f4141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.955+0000 7f44fa53b640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.955+0000 7f44fa53b640 1 monclient: found mon.noname-b 2024-01-21T04:33:50.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.955+0000 7f44fa53b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.955+0000 7f44fa53b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.955+0000 7f44fa53b640 1 -- 172.21.15.69:0/1822962835 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e400ae30 con 0x7f44f413b760 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 1 -- 172.21.15.69:0/3219509575 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140317c0 con 0x7f02240a9d10 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 10 monclient: dump: 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.026 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 1 -- 172.21.15.69:0/3219509575 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0214031920 con 0x7f02240a9d10 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 1 -- 172.21.15.69:0/3219509575 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214031cb0 con 0x7f02240a9d10 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f022b3b6640 5 monclient: authenticate success, global_id 4274 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f022b3b6640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f022b3b6640 4 set_mon_vals no callback set 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.956+0000 7f021bfff640 10 monclient: dump: 2024-01-21T04:33:50.027 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.028 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.028 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.028 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.028 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.028 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.028 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.028 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.028 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.028 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7ff32dd9b640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3280322a0 0x7ff3280c08b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff324000c00 tx=0x7ff3240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7ff31effd640 10 client.?.objecter ms_handle_connect 0x7ff3280322a0 2024-01-21T04:33:50.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7ff31effd640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7f022b3b6640 10 monclient: shutdown 2024-01-21T04:33:50.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7f022b3b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7f022b3b6640 1 -- 172.21.15.69:0/3219509575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02240a9d10 msgr2=0x7f022413f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7f022b3b6640 1 --2- 172.21.15.69:0/3219509575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02240a9d10 0x7f022413f080 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0214009870 tx=0x7f021402b860 comp rx=0 tx=0).stop 2024-01-21T04:33:50.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7f022b3b6640 1 -- 172.21.15.69:0/3219509575 shutdown_connections 2024-01-21T04:33:50.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7f022b3b6640 1 --2- 172.21.15.69:0/3219509575 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02240a9d10 0x7f022413f080 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7f022b3b6640 1 --2- 172.21.15.69:0/3219509575 >> v2:172.21.15.69:3300/0 conn(0x7f02240a8b20 0x7f0224145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7f022b3b6640 1 -- 172.21.15.69:0/3219509575 >> 172.21.15.69:0/3219509575 conn(0x7f0224066fb0 msgr2=0x7f02240b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7f022b3b6640 1 -- 172.21.15.69:0/3219509575 shutdown_connections 2024-01-21T04:33:50.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.957+0000 7f022b3b6640 1 -- 172.21.15.69:0/3219509575 wait complete. 2024-01-21T04:33:50.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f8c2086c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c1c0a8b20 0x7f8c1c142b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59690/0 (socket says 172.21.15.69:59690) 2024-01-21T04:33:50.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f8c2086c640 1 -- 172.21.15.69:0/3508552249 learned_addr learned my addr 172.21.15.69:0/3508552249 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f8c2086c640 10 monclient(hunting): get_auth_request con 0x7f8c1c0a8b20 auth_method 0 2024-01-21T04:33:50.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f8c2086c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f8c2086c640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f8c2086c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f022b3b6640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f827a526640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f826c0c8fb0 0x7f826c0b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f022b3b6640 10 monclient: monmap: 2024-01-21T04:33:50.033 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.033 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:50.033 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:50.033 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:50.033 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.033 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.033 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:50.033 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:50.033 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:50.033 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f022b3b6640 1 librados: starting msgr at 2024-01-21T04:33:50.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f022b3b6640 1 librados: starting objecter 2024-01-21T04:33:50.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f022b3b6640 1 Processor -- start 2024-01-21T04:33:50.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f8cbb59e640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cbc0b2020 0x7f8cbc1d81f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000c390 tx=0x7f8cb000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f8ca37fe640 10 client.?.objecter ms_handle_connect 0x7f8cbc0b2020 2024-01-21T04:33:50.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f8ca37fe640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f022b3b6640 1 -- start start 2024-01-21T04:33:50.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f022b3b6640 1 librados: setting wanted keys 2024-01-21T04:33:50.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f022b3b6640 1 librados: calling monclient init 2024-01-21T04:33:50.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.958+0000 7f022b3b6640 10 monclient: init 2024-01-21T04:33:50.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f827a526640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f826c0c8fb0 0x7f826c0b7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59698/0 (socket says 172.21.15.69:59698) 2024-01-21T04:33:50.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f827a526640 1 -- 172.21.15.69:0/2515081389 learned_addr learned my addr 172.21.15.69:0/2515081389 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f827a526640 10 monclient(hunting): get_auth_request con 0x7f826c0c8fb0 auth_method 0 2024-01-21T04:33:50.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f827a526640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f827a526640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f827a526640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T04:33:50.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02240a8b20 0x7f02241375c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 10 monclient(hunting): picked mon.noname-c con 0x7f02240a8b20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02240a9d10 0x7f0224142290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 10 monclient(hunting): picked mon.noname-b con 0x7f02240a9d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02240b2310 0x7f02241427d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 10 monclient(hunting): picked mon.noname-a con 0x7f02240b2310 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0224140e20 con 0x7f02240b2310 2024-01-21T04:33:50.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0224136490 con 0x7f02240a9d10 2024-01-21T04:33:50.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0224141b20 con 0x7f02240a8b20 2024-01-21T04:33:50.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f022b3b6640 10 monclient(hunting): authenticate will time out at 1388.301023s 2024-01-21T04:33:50.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f44fa53b640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44f413b760 0x7f44f4136850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f44ec00b9e0 tx=0x7f44ec00bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f44ea7fc640 10 client.?.objecter ms_handle_connect 0x7f44f413b760 2024-01-21T04:33:50.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.959+0000 7f44ea7fc640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.960+0000 7f8279524640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 0x7f826c0c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.961+0000 7f827a526640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.961+0000 7f827a526640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.961+0000 7f827a526640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.963+0000 7f8279524640 10 monclient(hunting): get_auth_request con 0x7f826c0c2e20 auth_method 0 2024-01-21T04:33:50.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.963+0000 7f8279524640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.963+0000 7f8279524640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.963+0000 7f8279524640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.963+0000 7f8c1bfff640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0a9d10 0x7f8c1c137e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.964+0000 7f8c1bfff640 10 monclient(hunting): get_auth_request con 0x7f8c1c0a9d10 auth_method 0 2024-01-21T04:33:50.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.964+0000 7f8c1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.964+0000 7f8c1bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.964+0000 7f8c1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.965+0000 7f8279524640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.965+0000 7f8279524640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.965+0000 7f8279524640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.966+0000 7f8c1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.966+0000 7f8c1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.966+0000 7f8c1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.966+0000 7f3b53fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b54143490 0x7f3b5413f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.967+0000 7f8c2086c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.967+0000 7f8c2086c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.967+0000 7f8c2086c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 1 -- 172.21.15.69:0/3774713728 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb00035f0 con 0x7f8cbc0b2020 2024-01-21T04:33:50.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:50.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 10 monclient: dump: 2024-01-21T04:33:50.043 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.043 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8cc111d640 5 monclient: authenticate success, global_id 4318 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 1 -- 172.21.15.69:0/3774713728 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb0003750 con 0x7f8cbc0b2020 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 1 -- 172.21.15.69:0/3774713728 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0003b00 con 0x7f8cbc0b2020 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8ca37fe640 10 monclient: dump: 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.968+0000 7f8cb959a640 4 set_mon_vals no callback set 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8cc111d640 10 monclient: _renew_subs 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8cc111d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8cc111d640 1 -- 172.21.15.69:0/3774713728 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cbc145730 con 0x7f8cbc0b2020 2024-01-21T04:33:50.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8cc111d640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8cc111d640 10 monclient: _renew_subs 2024-01-21T04:33:50.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8cc111d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8cc111d640 1 -- 172.21.15.69:0/3774713728 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cbc134b20 con 0x7f8cbc0b2020 2024-01-21T04:33:50.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8cc111d640 1 librados: init done 2024-01-21T04:33:50.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8279524640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-01-21T04:33:50.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8279524640 1 -- 172.21.15.69:0/2515081389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f826c0c8fb0 msgr2=0x7f826c0b7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8279524640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f826c0c8fb0 0x7f826c0b7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8279524640 1 -- 172.21.15.69:0/2515081389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826c035070 msgr2=0x7f826c0c8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8279524640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826c035070 0x7f826c0c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8279524640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8279524640 1 monclient: found mon.noname-c 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8279524640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8279524640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.969+0000 7f8279524640 1 -- 172.21.15.69:0/2515081389 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8264009460 con 0x7f826c0c2e20 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8cc111d640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8cc111d640 15 client.4318.objecter _session_command_op_assign -1 1 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8cc111d640 20 client.4318.objecter _get_session osd=-1 returning homeless 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8cc111d640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8cc111d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8cc111d640 1 -- 172.21.15.69:0/3774713728 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8cbc13fa80 con 0x7f8cbc0b2020 2024-01-21T04:33:50.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8c2086c640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2024-01-21T04:33:50.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8c2086c640 1 -- 172.21.15.69:0/3508552249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0a9d10 msgr2=0x7f8c1c137e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8c2086c640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c1c0a9d10 0x7f8c1c137e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8c2086c640 1 -- 172.21.15.69:0/3508552249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c1c0b2310 msgr2=0x7f8c1c140100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8c2086c640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c1c0b2310 0x7f8c1c140100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f0228951640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02240a9d10 0x7f0224142290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8c2086c640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8c2086c640 1 monclient: found mon.noname-b 2024-01-21T04:33:50.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8c2086c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8c2086c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.970+0000 7f8c2086c640 1 -- 172.21.15.69:0/3508552249 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c040094d0 con 0x7f8c1c0a8b20 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 1 -- 172.21.15.69:0/1822962835 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec00cb20 con 0x7f44f413b760 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 10 monclient: dump: 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.051 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 1 -- 172.21.15.69:0/1822962835 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec00cca0 con 0x7f44f413b760 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 1 -- 172.21.15.69:0/1822962835 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec014680 con 0x7f44f413b760 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ea7fc640 10 monclient: dump: 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44ebfff640 4 set_mon_vals no callback set 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44fad3c640 5 monclient: authenticate success, global_id 4321 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44fad3c640 10 monclient: _renew_subs 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44fad3c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44fad3c640 1 -- 172.21.15.69:0/1822962835 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f40aca50 con 0x7f44f413b760 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44fad3c640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44fad3c640 10 monclient: _renew_subs 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44fad3c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44fad3c640 1 -- 172.21.15.69:0/1822962835 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f4136f70 con 0x7f44f413b760 2024-01-21T04:33:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f44fad3c640 1 librados: init done 2024-01-21T04:33:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f0228951640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02240a9d10 0x7f0224142290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59728/0 (socket says 172.21.15.69:59728) 2024-01-21T04:33:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f0228951640 1 -- 172.21.15.69:0/417121987 learned_addr learned my addr 172.21.15.69:0/417121987 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.972+0000 7f0228951640 10 monclient(hunting): get_auth_request con 0x7f02240a9d10 auth_method 0 2024-01-21T04:33:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f0228951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f0228951640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f0228951640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f44fad3c640 10 client.4321.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f44fad3c640 15 client.4321.objecter _session_command_op_assign -1 1 2024-01-21T04:33:50.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f44fad3c640 20 client.4321.objecter _get_session osd=-1 returning homeless 2024-01-21T04:33:50.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f44fad3c640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f44fad3c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f44fad3c640 1 -- 172.21.15.69:0/1822962835 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f44f406eb20 con 0x7f44f413b760 2024-01-21T04:33:50.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f8c2086c640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c1c0a8b20 0x7f8c1c142b20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8c1400c410 tx=0x7f8c1400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f8c197fa640 10 client.?.objecter ms_handle_connect 0x7f8c1c0a8b20 2024-01-21T04:33:50.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f8c197fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f0228951640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f0228951640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.973+0000 7f0228951640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.975+0000 7f3b52ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b540aecb0 0x7f3b54141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.975+0000 7f3b53fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b54143490 0x7f3b5413f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46858/0 (socket says 172.21.15.69:46858) 2024-01-21T04:33:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.975+0000 7f3b53fff640 1 -- 172.21.15.69:0/776366037 learned_addr learned my addr 172.21.15.69:0/776366037 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.975+0000 7f3b53fff640 10 monclient(hunting): get_auth_request con 0x7f3b54143490 auth_method 0 2024-01-21T04:33:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.975+0000 7f3b53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.975+0000 7f3b53fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.975+0000 7f3b53fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.978+0000 7f0228951640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-01-21T04:33:50.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.978+0000 7f0228951640 1 -- 172.21.15.69:0/417121987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02240a8b20 msgr2=0x7f02241375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.978+0000 7f0228951640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02240a8b20 0x7f02241375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.978+0000 7f0228951640 1 -- 172.21.15.69:0/417121987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02240b2310 msgr2=0x7f02241427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.978+0000 7f0228951640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02240b2310 0x7f02241427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.978+0000 7f0228951640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.978+0000 7f0228951640 1 monclient: found mon.noname-b 2024-01-21T04:33:50.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.978+0000 7f0228951640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.978+0000 7f0228951640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.978+0000 7f0228951640 1 -- 172.21.15.69:0/417121987 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140094d0 con 0x7f02240a9d10 2024-01-21T04:33:50.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.979+0000 7f8279524640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 0x7f826c0c51c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8264004000 tx=0x7f82640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.979+0000 7f826a7fc640 10 client.?.objecter ms_handle_connect 0x7f826c0c2e20 2024-01-21T04:33:50.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.979+0000 7f826a7fc640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.979+0000 7f3b53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.979+0000 7f3b53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.979+0000 7f3b53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.983+0000 7f3b53fff640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-01-21T04:33:50.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.983+0000 7f3b53fff640 1 -- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b540aecb0 msgr2=0x7f3b54141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.983+0000 7f3b53fff640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b540aecb0 0x7f3b54141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.983+0000 7f3b53fff640 1 -- 172.21.15.69:0/776366037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b5406a8a0 msgr2=0x7f3b5413ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.983+0000 7f3b53fff640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b5406a8a0 0x7f3b5413ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.983+0000 7f3b53fff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.983+0000 7f3b53fff640 1 monclient: found mon.noname-c 2024-01-21T04:33:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.983+0000 7f3b53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.983+0000 7f3b53fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.983+0000 7f3b53fff640 1 -- 172.21.15.69:0/776366037 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b480094d0 con 0x7f3b54143490 2024-01-21T04:33:50.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 1 -- 172.21.15.69:0/3508552249 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c140035f0 con 0x7f8c1c0a8b20 2024-01-21T04:33:50.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 10 monclient: dump: 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 1 -- 172.21.15.69:0/3508552249 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c14003750 con 0x7f8c1c0a8b20 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 1 -- 172.21.15.69:0/3508552249 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c14003b40 con 0x7f8c1c0a8b20 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c197fa640 10 monclient: dump: 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.065 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c22ad0640 5 monclient: authenticate success, global_id 4327 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c22ad0640 10 monclient: _renew_subs 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c22ad0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c1affd640 4 set_mon_vals no callback set 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c22ad0640 1 -- 172.21.15.69:0/3508552249 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1c139d20 con 0x7f8c1c0a8b20 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c22ad0640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c22ad0640 10 monclient: _renew_subs 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c22ad0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c22ad0640 1 -- 172.21.15.69:0/3508552249 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1c138c30 con 0x7f8c1c0a8b20 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.985+0000 7f8c22ad0640 1 librados: init done 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.986+0000 7f8c22ad0640 10 client.4327.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.986+0000 7f8c22ad0640 15 client.4327.objecter _session_command_op_assign -1 1 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.986+0000 7f8c22ad0640 20 client.4327.objecter _get_session osd=-1 returning homeless 2024-01-21T04:33:50.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.986+0000 7f8c22ad0640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.986+0000 7f8c22ad0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.986+0000 7f8c22ad0640 1 -- 172.21.15.69:0/3508552249 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c1c0a9110 con 0x7f8c1c0a8b20 2024-01-21T04:33:50.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.988+0000 7f3b53fff640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b54143490 0x7f3b5413f5e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00b9a0 tx=0x7f3b4c00be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.988+0000 7f3b3bfff640 10 client.?.objecter ms_handle_connect 0x7f3b54143490 2024-01-21T04:33:50.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.988+0000 7f3b3bfff640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7f0228951640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02240a9d10 0x7f0224142290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0214000c00 tx=0x7f02140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7f0219ffb640 10 client.?.objecter ms_handle_connect 0x7f02240a9d10 2024-01-21T04:33:50.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7f0219ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 1 -- 172.21.15.69:0/4075847385 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32403aa60 con 0x7ff3280322a0 2024-01-21T04:33:50.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:50.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 10 monclient: dump: 2024-01-21T04:33:50.068 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.068 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.068 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 1 -- 172.21.15.69:0/4075847385 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32403abe0 con 0x7ff3280322a0 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 1 -- 172.21.15.69:0/4075847385 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3240443f0 con 0x7ff3280322a0 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:50.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff31effd640 10 monclient: dump: 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.989+0000 7ff32cd99640 4 set_mon_vals no callback set 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 5 monclient: authenticate success, global_id 4277 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 10 monclient: _renew_subs 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 1 -- 172.21.15.69:0/4075847385 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff328029e80 con 0x7ff3280322a0 2024-01-21T04:33:50.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 10 monclient: _renew_subs 2024-01-21T04:33:50.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 1 -- 172.21.15.69:0/4075847385 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32802a790 con 0x7ff3280322a0 2024-01-21T04:33:50.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 1 librados: init done 2024-01-21T04:33:50.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7fdc56ffd640 1 --2- 172.21.15.69:0/1225671724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5806ec10 0x7fdc5806f0a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdc50009d90 tx=0x7fdc5002bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29dcac650602326f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 10 client.4277.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 15 client.4277.objecter _session_command_op_assign -1 1 2024-01-21T04:33:50.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 20 client.4277.objecter _get_session osd=-1 returning homeless 2024-01-21T04:33:50.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.990+0000 7ff334d0f640 1 -- 172.21.15.69:0/4075847385 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff3280cd0d0 con 0x7ff3280322a0 2024-01-21T04:33:50.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 1 -- 172.21.15.69:0/2515081389 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264031830 con 0x7f826c0c2e20 2024-01-21T04:33:50.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:50.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 10 monclient: dump: 2024-01-21T04:33:50.073 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.073 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.073 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.073 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.073 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.073 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 1 -- 172.21.15.69:0/2515081389 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826403f070 con 0x7f826c0c2e20 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 1 -- 172.21.15.69:0/2515081389 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264031b90 con 0x7f826c0c2e20 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826a7fc640 10 monclient: dump: 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.074 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f826bfff640 4 set_mon_vals no callback set 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f827ad27640 5 monclient: authenticate success, global_id 4280 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f827ad27640 10 monclient: _renew_subs 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f827ad27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f827ad27640 1 -- 172.21.15.69:0/2515081389 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c0c3d00 con 0x7f826c0c2e20 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f827ad27640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f827ad27640 10 monclient: _renew_subs 2024-01-21T04:33:50.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f827ad27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f827ad27640 1 -- 172.21.15.69:0/2515081389 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c0c3a90 con 0x7f826c0c2e20 2024-01-21T04:33:50.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.991+0000 7f827ad27640 1 librados: init done 2024-01-21T04:33:50.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.992+0000 7f827ad27640 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.992+0000 7f827ad27640 15 client.4280.objecter _session_command_op_assign -1 1 2024-01-21T04:33:50.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.992+0000 7f827ad27640 20 client.4280.objecter _get_session osd=-1 returning homeless 2024-01-21T04:33:50.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.992+0000 7f827ad27640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.992+0000 7f827ad27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.992+0000 7f827ad27640 1 -- 172.21.15.69:0/2515081389 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f826c008710 con 0x7f826c0c2e20 2024-01-21T04:33:50.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 1 -- 172.21.15.69:0/776366037 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c00cb40 con 0x7f3b54143490 2024-01-21T04:33:50.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 10 monclient: dump: 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 1 -- 172.21.15.69:0/776366037 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c00cca0 con 0x7f3b54143490 2024-01-21T04:33:50.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 1 -- 172.21.15.69:0/776366037 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c014680 con 0x7f3b54143490 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b3bfff640 10 monclient: dump: 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b5a105640 5 monclient: authenticate success, global_id 4286 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b5a105640 10 monclient: _renew_subs 2024-01-21T04:33:50.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b5a105640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b5a105640 1 -- 172.21.15.69:0/776366037 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b54141030 con 0x7f3b54143490 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b5a105640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b5a105640 10 monclient: _renew_subs 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b5a105640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b5a105640 1 -- 172.21.15.69:0/776366037 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5413fb20 con 0x7f3b54143490 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b5a105640 1 librados: init done 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.995+0000 7f3b51ffb640 4 set_mon_vals no callback set 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.996+0000 7f3b5a105640 10 client.4286.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.996+0000 7f3b5a105640 15 client.4286.objecter _session_command_op_assign -1 1 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.996+0000 7f3b5a105640 20 client.4286.objecter _get_session osd=-1 returning homeless 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.996+0000 7f3b5a105640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.996+0000 7f3b5a105640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:49.996+0000 7f3b5a105640 1 -- 172.21.15.69:0/776366037 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3b34002910 con 0x7f3b54143490 2024-01-21T04:33:50.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 1 -- 172.21.15.69:0/1225671724 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5002cd80 con 0x7fdc5806ec10 2024-01-21T04:33:50.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 10 monclient: dump: 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 1 -- 172.21.15.69:0/1225671724 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc50034440 con 0x7fdc5806ec10 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc5cf6c640 5 monclient: authenticate success, global_id 4296 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc5cf6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc5cf6c640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 1 -- 172.21.15.69:0/1225671724 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc500347d0 con 0x7fdc5806ec10 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc5cf6c640 4 set_mon_vals no callback set 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.014+0000 7fdc567fc640 10 monclient: dump: 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.015+0000 7fdc5cf6c640 10 monclient: shutdown 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.015+0000 7fdc5cf6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.015+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1225671724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5806ec10 msgr2=0x7fdc5806f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.015+0000 7fdc5cf6c640 1 --2- 172.21.15.69:0/1225671724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5806ec10 0x7fdc5806f0a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdc50009d90 tx=0x7fdc5002bfd0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.015+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1225671724 shutdown_connections 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.015+0000 7fdc5cf6c640 1 --2- 172.21.15.69:0/1225671724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5806ec10 0x7fdc5806f0a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.015+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1225671724 >> 172.21.15.69:0/1225671724 conn(0x7fdc580b3960 msgr2=0x7fdc580b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.015+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1225671724 shutdown_connections 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.015+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1225671724 wait complete. 2024-01-21T04:33:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.016+0000 7fdc5cf6c640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.016+0000 7fdc5cf6c640 10 monclient: monmap: 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.016+0000 7fdc5cf6c640 1 librados: starting msgr at 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 1 librados: starting objecter 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 1 Processor -- start 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 1 -- start start 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 1 librados: setting wanted keys 2024-01-21T04:33:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 1 librados: calling monclient init 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 10 monclient: init 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5806ec10 0x7fdc580acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 10 monclient(hunting): picked mon.noname-a con 0x7fdc5806ec10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5806f5e0 0x7fdc581ebbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 10 monclient(hunting): picked mon.noname-c con 0x7fdc5806f5e0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc58071ab0 0x7fdc580890e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 10 monclient(hunting): picked mon.noname-b con 0x7fdc58071ab0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc581ec3e0 con 0x7fdc5806ec10 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc58160490 con 0x7fdc58071ab0 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5815d4e0 con 0x7fdc5806f5e0 2024-01-21T04:33:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.017+0000 7fdc5cf6c640 10 monclient(hunting): authenticate will time out at 1388.359182s 2024-01-21T04:33:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.021+0000 7f44ea7fc640 1 -- 172.21.15.69:0/1822962835 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f44ec01d3b0 con 0x7f44f413b760 2024-01-21T04:33:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7f44ea7fc640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f44d806c860 0x7f44d806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7f44f9d3a640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f44d806c860 0x7f44d806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7f44f9d3a640 10 monclient: get_auth_request con 0x7f44d806c860 auth_method 0 2024-01-21T04:33:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7fdc577fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5806ec10 0x7fdc580acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7fdc577fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5806ec10 0x7fdc580acb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53304/0 (socket says 172.21.15.69:53304) 2024-01-21T04:33:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7fdc577fe640 1 -- 172.21.15.69:0/1399184110 learned_addr learned my addr 172.21.15.69:0/1399184110 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7fdc577fe640 10 monclient(hunting): get_auth_request con 0x7fdc5806ec10 auth_method 0 2024-01-21T04:33:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7fdc577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7fdc577fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7fdc577fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7f44ea7fc640 10 client.4321.objecter ms_handle_connect 0x7f44d806c860 2024-01-21T04:33:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.022+0000 7f44f9d3a640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f44d806c860 0x7f44d806ed20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f44f406e870 tx=0x7f44e4031000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.023+0000 7fdc577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.023+0000 7fdc577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.023+0000 7fdc577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.026+0000 7fdc577fe640 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.026+0000 7fdc577fe640 1 -- 172.21.15.69:0/1399184110 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5806f5e0 msgr2=0x7fdc581ebbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.026+0000 7fdc577fe640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc5806f5e0 0x7fdc581ebbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.026+0000 7fdc577fe640 1 -- 172.21.15.69:0/1399184110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc58071ab0 msgr2=0x7fdc580890e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.026+0000 7fdc577fe640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc58071ab0 0x7fdc580890e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.026+0000 7fdc577fe640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.026+0000 7fdc577fe640 1 monclient: found mon.noname-a 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.026+0000 7fdc577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.026+0000 7fdc577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.026+0000 7fdc577fe640 1 -- 172.21.15.69:0/1399184110 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc50009a70 con 0x7fdc5806ec10 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.028+0000 7fdc577fe640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5806ec10 0x7fdc580acb90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00c430 tx=0x7fdc4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.028+0000 7fdc3ffff640 10 client.?.objecter ms_handle_connect 0x7fdc5806ec10 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.028+0000 7fdc3ffff640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 1 -- 172.21.15.69:0/1399184110 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c0052d0 con 0x7fdc5806ec10 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 10 monclient: dump: 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 1 -- 172.21.15.69:0/1399184110 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4c005430 con 0x7fdc5806ec10 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 1 -- 172.21.15.69:0/1399184110 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c003830 con 0x7fdc5806ec10 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc3ffff640 10 monclient: dump: 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.094 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.031+0000 7fdc55ffb640 4 set_mon_vals no callback set 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.032+0000 7fdc5cf6c640 5 monclient: authenticate success, global_id 4302 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.032+0000 7fdc5cf6c640 10 monclient: _renew_subs 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.032+0000 7fdc5cf6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.032+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1399184110 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc5815d700 con 0x7fdc5806ec10 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.032+0000 7fdc5cf6c640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.032+0000 7fdc5cf6c640 10 monclient: _renew_subs 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.032+0000 7fdc5cf6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.032+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1399184110 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc58089800 con 0x7fdc5806ec10 2024-01-21T04:33:50.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.032+0000 7fdc5cf6c640 1 librados: init done 2024-01-21T04:33:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.033+0000 7fdc5cf6c640 10 client.4302.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.033+0000 7fdc5cf6c640 15 client.4302.objecter _session_command_op_assign -1 1 2024-01-21T04:33:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.033+0000 7fdc5cf6c640 20 client.4302.objecter _get_session osd=-1 returning homeless 2024-01-21T04:33:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.033+0000 7fdc5cf6c640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.033+0000 7fdc5cf6c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.033+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1399184110 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdc58071320 con 0x7fdc5806ec10 2024-01-21T04:33:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.036+0000 7f8c197fa640 1 -- 172.21.15.69:0/3508552249 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8c14003ca0 con 0x7f8c1c0a8b20 2024-01-21T04:33:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.036+0000 7f8c197fa640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8bfc06c860 0x7f8bfc06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.037+0000 7f8c1bfff640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8bfc06c860 0x7f8bfc06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.037+0000 7f8c1bfff640 10 monclient: get_auth_request con 0x7f8bfc06c860 auth_method 0 2024-01-21T04:33:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.037+0000 7f8c197fa640 10 client.4327.objecter ms_handle_connect 0x7f8bfc06c860 2024-01-21T04:33:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.037+0000 7f8c1bfff640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8bfc06c860 0x7f8bfc06ed20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8c04036000 tx=0x7f8c0403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.038+0000 7f8c197fa640 1 -- 172.21.15.69:0/3508552249 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8c140010c0 con 0x7f8c1c0a8b20 2024-01-21T04:33:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.038+0000 7f8c197fa640 10 client.4327.objecter ms_dispatch 0x7f8c1c000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T04:33:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.038+0000 7f8c197fa640 3 client.4327.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.038+0000 7f8c197fa640 3 client.4327.objecter handle_osd_map decoding full epoch 25 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 10 client.4327.objecter checking command 1 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] conn(0x7f8bfc072c40 0x7f8bfc075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 20 client.4327.objecter _get_session s=0x7f8bfc072570 osd=5 3 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 20 client.4327.objecter put_session s=0x7f8bfc072570 osd=5 3 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 15 client.4327.objecter _session_command_op_remove -1 1 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 20 client.4327.objecter _get_session s=0x7f8bfc072570 osd=5 3 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 20 client.4327.objecter get_session s=0x7f8bfc072570 osd=5 3 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 15 client.4327.objecter _session_command_op_assign 5 1 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 20 client.4327.objecter put_session s=0x7f8bfc072570 osd=5 4 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 10 client.4327.objecter _send_command 1 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 1 -- 172.21.15.69:0/3508552249 --> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8bfc075710 con 0x7f8bfc072c40 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c197fa640 20 client.4327.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c2106d640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] conn(0x7f8bfc072c40 0x7f8bfc075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 1 -- 172.21.15.69:0/417121987 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140328e0 con 0x7f02240a9d10 2024-01-21T04:33:50.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:50.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 10 monclient: dump: 2024-01-21T04:33:50.099 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.099 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 1 -- 172.21.15.69:0/417121987 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0214032a60 con 0x7f02240a9d10 2024-01-21T04:33:50.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f022b3b6640 5 monclient: authenticate success, global_id 4330 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f022b3b6640 10 monclient: _renew_subs 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f022b3b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f022b3b6640 1 -- 172.21.15.69:0/417121987 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0224140140 con 0x7f02240a9d10 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 1 -- 172.21.15.69:0/417121987 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214032e10 con 0x7f02240a9d10 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f0219ffb640 10 monclient: dump: 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.101 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.102 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.102 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f8c2106d640 10 monclient: get_auth_request con 0x7f8bfc072c40 auth_method 0 2024-01-21T04:33:50.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.039+0000 7f021b7fe640 4 set_mon_vals no callback set 2024-01-21T04:33:50.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.040+0000 7f022b3b6640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.040+0000 7f022b3b6640 10 monclient: _renew_subs 2024-01-21T04:33:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.040+0000 7f022b3b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.040+0000 7f022b3b6640 1 -- 172.21.15.69:0/417121987 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0224142ef0 con 0x7f02240a9d10 2024-01-21T04:33:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.040+0000 7f022b3b6640 1 librados: init done 2024-01-21T04:33:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.040+0000 7f022b3b6640 10 client.4330.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.040+0000 7f022b3b6640 15 client.4330.objecter _session_command_op_assign -1 1 2024-01-21T04:33:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.040+0000 7f022b3b6640 20 client.4330.objecter _get_session osd=-1 returning homeless 2024-01-21T04:33:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.040+0000 7f022b3b6640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.040+0000 7f022b3b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.040+0000 7f022b3b6640 1 -- 172.21.15.69:0/417121987 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f02240a9110 con 0x7f02240a9d10 2024-01-21T04:33:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.043+0000 7f8c197fa640 10 client.4327.objecter ms_handle_connect 0x7f8bfc072c40 2024-01-21T04:33:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.043+0000 7f8c2106d640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] conn(0x7f8bfc072c40 0x7f8bfc075060 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.052+0000 7f8ca37fe640 1 -- 172.21.15.69:0/3774713728 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8cb0003c60 con 0x7f8cbc0b2020 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.053+0000 7f8ca37fe640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8ca806c8b0 0x7f8ca806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.053+0000 7f8cbad9d640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8ca806c8b0 0x7f8ca806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.053+0000 7f8cbad9d640 10 monclient: get_auth_request con 0x7f8ca806c8b0 auth_method 0 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.054+0000 7f8cbad9d640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8ca806c8b0 0x7f8ca806ed70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4036000 tx=0x7f8cb402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.054+0000 7f8ca37fe640 10 client.4318.objecter ms_handle_connect 0x7f8ca806c8b0 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.054+0000 7f8ca37fe640 1 -- 172.21.15.69:0/3774713728 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8cb00585e0 con 0x7f8cbc0b2020 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.054+0000 7f8ca37fe640 10 client.4318.objecter ms_dispatch 0x7f8cbc000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.054+0000 7f8ca37fe640 3 client.4318.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.054+0000 7f8ca37fe640 3 client.4318.objecter handle_osd_map decoding full epoch 25 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 10 client.4318.objecter checking command 1 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] conn(0x7f8ca8072c90 0x7f8ca80750b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 20 client.4318.objecter _get_session s=0x7f8ca80725c0 osd=4 3 2024-01-21T04:33:50.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 20 client.4318.objecter put_session s=0x7f8ca80725c0 osd=4 3 2024-01-21T04:33:50.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 15 client.4318.objecter _session_command_op_remove -1 1 2024-01-21T04:33:50.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 20 client.4318.objecter _get_session s=0x7f8ca80725c0 osd=4 3 2024-01-21T04:33:50.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 20 client.4318.objecter get_session s=0x7f8ca80725c0 osd=4 3 2024-01-21T04:33:50.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 15 client.4318.objecter _session_command_op_assign 4 1 2024-01-21T04:33:50.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 20 client.4318.objecter put_session s=0x7f8ca80725c0 osd=4 4 2024-01-21T04:33:50.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 10 client.4318.objecter _send_command 1 2024-01-21T04:33:50.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 1 -- 172.21.15.69:0/3774713728 --> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ca8075760 con 0x7f8ca8072c90 2024-01-21T04:33:50.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8ca37fe640 20 client.4318.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7f8cba59c640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] conn(0x7f8ca8072c90 0x7f8ca80750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.055+0000 7fdc3ffff640 1 -- 172.21.15.69:0/1399184110 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fdc4c003990 con 0x7fdc5806ec10 2024-01-21T04:33:50.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.056+0000 7f8cba59c640 10 monclient: get_auth_request con 0x7f8ca8072c90 auth_method 0 2024-01-21T04:33:50.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.056+0000 7fdc3ffff640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdc4406c8b0 0x7fdc4406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.056+0000 7fdc56ffd640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdc4406c8b0 0x7fdc4406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.056+0000 7fdc56ffd640 10 monclient: get_auth_request con 0x7fdc4406c8b0 auth_method 0 2024-01-21T04:33:50.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.056+0000 7fdc56ffd640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdc4406c8b0 0x7fdc4406ed70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdc50036000 tx=0x7fdc5002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.056+0000 7fdc3ffff640 10 client.4302.objecter ms_handle_connect 0x7fdc4406c8b0 2024-01-21T04:33:50.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 1 -- 172.21.15.69:0/1822962835 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f44ec058f30 con 0x7f44f413b760 2024-01-21T04:33:50.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 10 client.4321.objecter ms_dispatch 0x7f44f4000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T04:33:50.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 3 client.4321.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T04:33:50.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 3 client.4321.objecter handle_osd_map decoding full epoch 25 2024-01-21T04:33:50.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 10 client.4321.objecter checking command 1 2024-01-21T04:33:50.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] conn(0x7f44d8072c40 0x7f44d8075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 20 client.4321.objecter _get_session s=0x7f44d8072570 osd=0 3 2024-01-21T04:33:50.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 20 client.4321.objecter put_session s=0x7f44d8072570 osd=0 3 2024-01-21T04:33:50.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 15 client.4321.objecter _session_command_op_remove -1 1 2024-01-21T04:33:50.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 20 client.4321.objecter _get_session s=0x7f44d8072570 osd=0 3 2024-01-21T04:33:50.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 20 client.4321.objecter get_session s=0x7f44d8072570 osd=0 3 2024-01-21T04:33:50.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 15 client.4321.objecter _session_command_op_assign 0 1 2024-01-21T04:33:50.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 20 client.4321.objecter put_session s=0x7f44d8072570 osd=0 4 2024-01-21T04:33:50.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 10 client.4321.objecter _send_command 1 2024-01-21T04:33:50.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 1 -- 172.21.15.69:0/1822962835 --> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f44d8075710 con 0x7f44d8072c40 2024-01-21T04:33:50.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44ea7fc640 20 client.4321.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 1 -- 172.21.15.69:0/1399184110 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdc4c058530 con 0x7fdc5806ec10 2024-01-21T04:33:50.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 10 client.4302.objecter ms_dispatch 0x7fdc58000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T04:33:50.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 3 client.4302.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T04:33:50.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 3 client.4302.objecter handle_osd_map decoding full epoch 25 2024-01-21T04:33:50.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 10 client.4302.objecter checking command 1 2024-01-21T04:33:50.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x7fdc44072c90 0x7fdc440750b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 20 client.4302.objecter _get_session s=0x7fdc440725c0 osd=1 3 2024-01-21T04:33:50.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 20 client.4302.objecter put_session s=0x7fdc440725c0 osd=1 3 2024-01-21T04:33:50.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 15 client.4302.objecter _session_command_op_remove -1 1 2024-01-21T04:33:50.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 20 client.4302.objecter _get_session s=0x7fdc440725c0 osd=1 3 2024-01-21T04:33:50.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 20 client.4302.objecter get_session s=0x7fdc440725c0 osd=1 3 2024-01-21T04:33:50.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 15 client.4302.objecter _session_command_op_assign 1 1 2024-01-21T04:33:50.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 20 client.4302.objecter put_session s=0x7fdc440725c0 osd=1 4 2024-01-21T04:33:50.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 10 client.4302.objecter _send_command 1 2024-01-21T04:33:50.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 1 -- 172.21.15.69:0/1399184110 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc44075760 con 0x7fdc44072c90 2024-01-21T04:33:50.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7fdc3ffff640 20 client.4302.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f8ca37fe640 10 client.4318.objecter ms_handle_connect 0x7f8ca8072c90 2024-01-21T04:33:50.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f8cba59c640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] conn(0x7f8ca8072c90 0x7f8ca80750b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.059+0000 7f44f9539640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] conn(0x7f44d8072c40 0x7f44d8075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.060+0000 7f44f9539640 10 monclient: get_auth_request con 0x7f44d8072c40 auth_method 0 2024-01-21T04:33:50.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.060+0000 7fdc57fff640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x7fdc44072c90 0x7fdc440750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.060+0000 7fdc57fff640 10 monclient: get_auth_request con 0x7fdc44072c90 auth_method 0 2024-01-21T04:33:50.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.062+0000 7fdc3ffff640 1 -- 172.21.15.69:0/1399184110 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc4c001140 con 0x7fdc5806ec10 2024-01-21T04:33:50.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.062+0000 7fdc3ffff640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-01-21T04:33:50.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.063+0000 7fdc57fff640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x7fdc44072c90 0x7fdc440750b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.063+0000 7fdc3ffff640 10 client.4302.objecter ms_handle_connect 0x7fdc44072c90 2024-01-21T04:33:50.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.064+0000 7f44f9539640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] conn(0x7f44d8072c40 0x7f44d8075060 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.064+0000 7f44ea7fc640 10 client.4321.objecter ms_handle_connect 0x7f44d8072c40 2024-01-21T04:33:50.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.088+0000 7f8c197fa640 1 -- 172.21.15.69:0/3508552249 <== osd.5 v2:172.21.15.144:6802/1782989653 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8bfc075710 con 0x7f8bfc072c40 2024-01-21T04:33:50.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.088+0000 7f8c197fa640 10 client.4327.objecter ms_dispatch 0x7f8c1c000b90 command_reply(tid 1: 0 ) v1 2024-01-21T04:33:50.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.088+0000 7f8c197fa640 10 client.4327.objecter _finish_command 1 = system:0 2024-01-21T04:33:50.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.088+0000 7f8c197fa640 20 client.4327.objecter put_session s=0x7f8bfc072570 osd=5 4 2024-01-21T04:33:50.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.088+0000 7f8c197fa640 15 client.4327.objecter _session_command_op_remove 5 1 2024-01-21T04:33:50.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.090+0000 7f3b3bfff640 1 -- 172.21.15.69:0/776366037 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f3b4c01d3b0 con 0x7f3b54143490 2024-01-21T04:33:50.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.091+0000 7f3b3bfff640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f3b4006c860 0x7f3b4006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.091+0000 7f3b537fe640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f3b4006c860 0x7f3b4006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.091+0000 7f3b537fe640 10 monclient: get_auth_request con 0x7f3b4006c860 auth_method 0 2024-01-21T04:33:50.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.092+0000 7f3b3bfff640 10 client.4286.objecter ms_handle_connect 0x7f3b4006c860 2024-01-21T04:33:50.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.092+0000 7f3b537fe640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f3b4006c860 0x7f3b4006ed20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3b44001560 tx=0x7f3b44007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 1 -- 172.21.15.69:0/776366037 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3b4c0589a0 con 0x7f3b54143490 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 10 client.4286.objecter ms_dispatch 0x7f3b54000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 3 client.4286.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 3 client.4286.objecter handle_osd_map decoding full epoch 25 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 10 client.4286.objecter checking command 1 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] conn(0x7f3b40072c40 0x7f3b40075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 20 client.4286.objecter _get_session s=0x7f3b40072570 osd=6 3 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 20 client.4286.objecter put_session s=0x7f3b40072570 osd=6 3 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 15 client.4286.objecter _session_command_op_remove -1 1 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 20 client.4286.objecter _get_session s=0x7f3b40072570 osd=6 3 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 20 client.4286.objecter get_session s=0x7f3b40072570 osd=6 3 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 15 client.4286.objecter _session_command_op_assign 6 1 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 20 client.4286.objecter put_session s=0x7f3b40072570 osd=6 4 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 10 client.4286.objecter _send_command 1 2024-01-21T04:33:50.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 1 -- 172.21.15.69:0/776366037 --> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b40075710 con 0x7f3b40072c40 2024-01-21T04:33:50.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.095+0000 7f3b3bfff640 20 client.4286.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.096+0000 7f3b52ffd640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] conn(0x7f3b40072c40 0x7f3b40075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.097+0000 7f3b52ffd640 10 monclient: get_auth_request con 0x7f3b40072c40 auth_method 0 2024-01-21T04:33:50.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.101+0000 7f3b3bfff640 10 client.4286.objecter ms_handle_connect 0x7f3b40072c40 2024-01-21T04:33:50.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.101+0000 7f3b52ffd640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] conn(0x7f3b40072c40 0x7f3b40075060 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.103+0000 7f8c22ad0640 10 client.4327.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T04:33:50.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.103+0000 7f8c22ad0640 15 client.4327.objecter _session_command_op_assign -1 2 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.103+0000 7f8c22ad0640 20 client.4327.objecter _get_session s=0x7f8bfc072570 osd=5 3 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.103+0000 7f8c22ad0640 20 client.4327.objecter put_session s=0x7f8bfc072570 osd=5 3 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.103+0000 7f8c22ad0640 20 client.4327.objecter _get_session s=0x7f8bfc072570 osd=5 3 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.103+0000 7f8c22ad0640 15 client.4327.objecter _session_command_op_remove -1 2 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.103+0000 7f8c22ad0640 20 client.4327.objecter get_session s=0x7f8bfc072570 osd=5 3 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.103+0000 7f8c22ad0640 15 client.4327.objecter _session_command_op_assign 5 2 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.103+0000 7f8c22ad0640 20 client.4327.objecter put_session s=0x7f8bfc072570 osd=5 4 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.103+0000 7f8c22ad0640 10 client.4327.objecter _send_command 2 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.103+0000 7f8c22ad0640 1 -- 172.21.15.69:0/3508552249 --> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8c1c138700 con 0x7f8bfc072c40 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.109+0000 7f8c197fa640 1 -- 172.21.15.69:0/3508552249 <== osd.5 v2:172.21.15.144:6802/1782989653 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8c0c00a720 con 0x7f8bfc072c40 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.109+0000 7f8c197fa640 10 client.4327.objecter ms_dispatch 0x7f8c1c000b90 command_reply(tid 2: 0 ) v1 2024-01-21T04:33:50.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.109+0000 7f8c197fa640 10 client.4327.objecter _finish_command 2 = system:0 2024-01-21T04:33:50.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.109+0000 7f8c197fa640 20 client.4327.objecter put_session s=0x7f8bfc072570 osd=5 4 2024-01-21T04:33:50.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.109+0000 7f8c197fa640 15 client.4327.objecter _session_command_op_remove 5 2 2024-01-21T04:33:50.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.109+0000 7f8c22ad0640 10 librados: watch_flush enter 2024-01-21T04:33:50.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.109+0000 7f8c22ad0640 10 librados: watch_flush exit 2024-01-21T04:33:50.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 10 client.4327.objecter close_session for osd.5 2024-01-21T04:33:50.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 1 -- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] conn(0x7f8bfc072c40 msgr2=0x7f8bfc075060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] conn(0x7f8bfc072c40 0x7f8bfc075060 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 20 client.4327.objecter put_session s=0x7f8bfc072570 osd=5 1 2024-01-21T04:33:50.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 20 client.4327.objecter shutdown clearing up homeless session... 2024-01-21T04:33:50.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 10 client.4327.objecter successfully canceled tick 2024-01-21T04:33:50.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 1 -- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8bfc06c860 msgr2=0x7f8bfc06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8bfc06c860 0x7f8bfc06ed20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8c04036000 tx=0x7f8c0403c040 comp rx=0 tx=0).stop 2024-01-21T04:33:50.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 10 monclient: shutdown 2024-01-21T04:33:50.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 20 monclient: shutdown canceling and discarding version request 1 2024-01-21T04:33:50.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 1 -- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c1c0a8b20 msgr2=0x7f8c1c142b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 1 --2- 172.21.15.69:0/3508552249 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c1c0a8b20 0x7f8c1c142b20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8c1400c410 tx=0x7f8c1400c8d0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c2106d640 1 -- 172.21.15.69:0/3508552249 reap_dead start 2024-01-21T04:33:50.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 1 -- 172.21.15.69:0/3508552249 shutdown_connections 2024-01-21T04:33:50.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 1 -- 172.21.15.69:0/3508552249 >> 172.21.15.69:0/3508552249 conn(0x7f8c1c066fb0 msgr2=0x7f8c1c0b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 1 -- 172.21.15.69:0/3508552249 shutdown_connections 2024-01-21T04:33:50.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 1 -- 172.21.15.69:0/3508552249 wait complete. 2024-01-21T04:33:50.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8ca37fe640 1 -- 172.21.15.69:0/3774713728 <== osd.4 v2:172.21.15.144:6826/3897583001 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8ca8075760 con 0x7f8ca8072c90 2024-01-21T04:33:50.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8ca37fe640 10 client.4318.objecter ms_dispatch 0x7f8cbc000b90 command_reply(tid 1: 0 ) v1 2024-01-21T04:33:50.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8ca37fe640 10 client.4318.objecter _finish_command 1 = system:0 2024-01-21T04:33:50.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8ca37fe640 20 client.4318.objecter put_session s=0x7f8ca80725c0 osd=4 4 2024-01-21T04:33:50.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8ca37fe640 15 client.4318.objecter _session_command_op_remove 4 1 2024-01-21T04:33:50.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.110+0000 7f8c22ad0640 1 librados: shutdown 2024-01-21T04:33:50.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.113+0000 7f44ea7fc640 1 -- 172.21.15.69:0/1822962835 <== osd.0 v2:172.21.15.69:6824/335500148 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f44d8075710 con 0x7f44d8072c40 2024-01-21T04:33:50.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.113+0000 7f44ea7fc640 10 client.4321.objecter ms_dispatch 0x7f44f4000b90 command_reply(tid 1: 0 ) v1 2024-01-21T04:33:50.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.113+0000 7f44ea7fc640 10 client.4321.objecter _finish_command 1 = system:0 2024-01-21T04:33:50.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.113+0000 7f44ea7fc640 20 client.4321.objecter put_session s=0x7f44d8072570 osd=0 4 2024-01-21T04:33:50.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.113+0000 7f44ea7fc640 15 client.4321.objecter _session_command_op_remove 0 1 2024-01-21T04:33:50.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.115+0000 7fdc3ffff640 1 -- 172.21.15.69:0/1399184110 <== osd.1 v2:172.21.15.69:6808/3868480563 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc44075760 con 0x7fdc44072c90 2024-01-21T04:33:50.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.115+0000 7fdc3ffff640 10 client.4302.objecter ms_dispatch 0x7fdc58000b90 command_reply(tid 1: 0 ) v1 2024-01-21T04:33:50.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.115+0000 7fdc3ffff640 10 client.4302.objecter _finish_command 1 = system:0 2024-01-21T04:33:50.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.115+0000 7fdc3ffff640 20 client.4302.objecter put_session s=0x7fdc440725c0 osd=1 4 2024-01-21T04:33:50.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.115+0000 7fdc3ffff640 15 client.4302.objecter _session_command_op_remove 1 1 2024-01-21T04:33:50.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.117+0000 7f8ca37fe640 1 -- 172.21.15.69:0/3774713728 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cb00010c0 con 0x7f8cbc0b2020 2024-01-21T04:33:50.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.117+0000 7f8ca37fe640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.118+0000 7ff31effd640 1 -- 172.21.15.69:0/4075847385 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff32404d450 con 0x7ff3280322a0 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.118+0000 7ff31effd640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7ff31006c860 0x7ff31006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.119+0000 7ff32e59c640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7ff31006c860 0x7ff31006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.119+0000 7ff32e59c640 10 monclient: get_auth_request con 0x7ff31006c860 auth_method 0 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.119+0000 7ff31effd640 10 client.4277.objecter ms_handle_connect 0x7ff31006c860 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.119+0000 7ff32e59c640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7ff31006c860 0x7ff31006ed20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff320001560 tx=0x7ff320007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 1 -- 172.21.15.69:0/4075847385 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7ff32407b1e0 con 0x7ff3280322a0 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 10 client.4277.objecter ms_dispatch 0x7ff330000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 3 client.4277.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 3 client.4277.objecter handle_osd_map decoding full epoch 25 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 10 client.4277.objecter checking command 1 2024-01-21T04:33:50.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] conn(0x7ff310072c40 0x7ff310075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 20 client.4277.objecter _get_session s=0x7ff310072570 osd=3 3 2024-01-21T04:33:50.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 20 client.4277.objecter put_session s=0x7ff310072570 osd=3 3 2024-01-21T04:33:50.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 15 client.4277.objecter _session_command_op_remove -1 1 2024-01-21T04:33:50.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 20 client.4277.objecter _get_session s=0x7ff310072570 osd=3 3 2024-01-21T04:33:50.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 20 client.4277.objecter get_session s=0x7ff310072570 osd=3 3 2024-01-21T04:33:50.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 15 client.4277.objecter _session_command_op_assign 3 1 2024-01-21T04:33:50.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 20 client.4277.objecter put_session s=0x7ff310072570 osd=3 4 2024-01-21T04:33:50.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 10 client.4277.objecter _send_command 1 2024-01-21T04:33:50.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 1 -- 172.21.15.69:0/4075847385 --> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff310075710 con 0x7ff310072c40 2024-01-21T04:33:50.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.122+0000 7ff31effd640 20 client.4277.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.123+0000 7ff32ed9d640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] conn(0x7ff310072c40 0x7ff310075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.128 INFO:teuthology.orchestra.run.smithi069.stdout:55834574854 2024-01-21T04:33:50.128 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-21T04:33:50.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.124+0000 7ff32ed9d640 10 monclient: get_auth_request con 0x7ff310072c40 auth_method 0 2024-01-21T04:33:50.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.128+0000 7f44ea7fc640 1 -- 172.21.15.69:0/1822962835 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44ec054b70 con 0x7f44f413b760 2024-01-21T04:33:50.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.128+0000 7f44ea7fc640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-01-21T04:33:50.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.131+0000 7f8cc111d640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T04:33:50.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.131+0000 7f8cc111d640 15 client.4318.objecter _session_command_op_assign -1 2 2024-01-21T04:33:50.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.131+0000 7f8cc111d640 20 client.4318.objecter _get_session s=0x7f8ca80725c0 osd=4 3 2024-01-21T04:33:50.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.131+0000 7f8cc111d640 20 client.4318.objecter put_session s=0x7f8ca80725c0 osd=4 3 2024-01-21T04:33:50.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.131+0000 7f8cc111d640 20 client.4318.objecter _get_session s=0x7f8ca80725c0 osd=4 3 2024-01-21T04:33:50.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.131+0000 7f8cc111d640 15 client.4318.objecter _session_command_op_remove -1 2 2024-01-21T04:33:50.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.131+0000 7f8cc111d640 20 client.4318.objecter get_session s=0x7f8ca80725c0 osd=4 3 2024-01-21T04:33:50.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.131+0000 7f8cc111d640 15 client.4318.objecter _session_command_op_assign 4 2 2024-01-21T04:33:50.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.131+0000 7f8cc111d640 20 client.4318.objecter put_session s=0x7f8ca80725c0 osd=4 4 2024-01-21T04:33:50.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.131+0000 7f8cc111d640 10 client.4318.objecter _send_command 2 2024-01-21T04:33:50.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.131+0000 7f8cc111d640 1 -- 172.21.15.69:0/3774713728 --> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8cbc1d7b00 con 0x7f8ca8072c90 2024-01-21T04:33:50.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.132+0000 7ff32ed9d640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] conn(0x7ff310072c40 0x7ff310075060 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.132+0000 7ff31effd640 10 client.4277.objecter ms_handle_connect 0x7ff310072c40 2024-01-21T04:33:50.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.133+0000 7fdc5cf6c640 10 client.4302.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T04:33:50.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.133+0000 7fdc5cf6c640 15 client.4302.objecter _session_command_op_assign -1 2 2024-01-21T04:33:50.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.133+0000 7fdc5cf6c640 20 client.4302.objecter _get_session s=0x7fdc440725c0 osd=1 3 2024-01-21T04:33:50.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.133+0000 7fdc5cf6c640 20 client.4302.objecter put_session s=0x7fdc440725c0 osd=1 3 2024-01-21T04:33:50.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.133+0000 7fdc5cf6c640 20 client.4302.objecter _get_session s=0x7fdc440725c0 osd=1 3 2024-01-21T04:33:50.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.133+0000 7fdc5cf6c640 15 client.4302.objecter _session_command_op_remove -1 2 2024-01-21T04:33:50.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.133+0000 7fdc5cf6c640 20 client.4302.objecter get_session s=0x7fdc440725c0 osd=1 3 2024-01-21T04:33:50.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.133+0000 7fdc5cf6c640 15 client.4302.objecter _session_command_op_assign 1 2 2024-01-21T04:33:50.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.133+0000 7fdc5cf6c640 20 client.4302.objecter put_session s=0x7fdc440725c0 osd=1 4 2024-01-21T04:33:50.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.133+0000 7fdc5cf6c640 10 client.4302.objecter _send_command 2 2024-01-21T04:33:50.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.133+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1399184110 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdc58071080 con 0x7fdc44072c90 2024-01-21T04:33:50.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.135+0000 7f44fad3c640 10 client.4321.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T04:33:50.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.135+0000 7f44fad3c640 15 client.4321.objecter _session_command_op_assign -1 2 2024-01-21T04:33:50.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.135+0000 7f44fad3c640 20 client.4321.objecter _get_session s=0x7f44d8072570 osd=0 3 2024-01-21T04:33:50.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.135+0000 7f44fad3c640 20 client.4321.objecter put_session s=0x7f44d8072570 osd=0 3 2024-01-21T04:33:50.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.135+0000 7f44fad3c640 20 client.4321.objecter _get_session s=0x7f44d8072570 osd=0 3 2024-01-21T04:33:50.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.135+0000 7f44fad3c640 15 client.4321.objecter _session_command_op_remove -1 2 2024-01-21T04:33:50.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.135+0000 7f44fad3c640 20 client.4321.objecter get_session s=0x7f44d8072570 osd=0 3 2024-01-21T04:33:50.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.135+0000 7f44fad3c640 15 client.4321.objecter _session_command_op_assign 0 2 2024-01-21T04:33:50.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.135+0000 7f44fad3c640 20 client.4321.objecter put_session s=0x7f44d8072570 osd=0 4 2024-01-21T04:33:50.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.135+0000 7f44fad3c640 10 client.4321.objecter _send_command 2 2024-01-21T04:33:50.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.135+0000 7f44fad3c640 1 -- 172.21.15.69:0/1822962835 --> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f44f4070fe0 con 0x7f44d8072c40 2024-01-21T04:33:50.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.139+0000 7f8ca37fe640 1 -- 172.21.15.69:0/3774713728 <== osd.4 v2:172.21.15.144:6826/3897583001 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8cbc1d7b00 con 0x7f8ca8072c90 2024-01-21T04:33:50.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.139+0000 7f8ca37fe640 10 client.4318.objecter ms_dispatch 0x7f8cbc000b90 command_reply(tid 2: 0 ) v1 2024-01-21T04:33:50.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.139+0000 7f8ca37fe640 10 client.4318.objecter _finish_command 2 = system:0 2024-01-21T04:33:50.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.139+0000 7f8ca37fe640 20 client.4318.objecter put_session s=0x7f8ca80725c0 osd=4 4 2024-01-21T04:33:50.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.139+0000 7f8ca37fe640 15 client.4318.objecter _session_command_op_remove 4 2 2024-01-21T04:33:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.139+0000 7f8cc111d640 10 librados: watch_flush enter 2024-01-21T04:33:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.139+0000 7f8cc111d640 10 librados: watch_flush exit 2024-01-21T04:33:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.139+0000 7f8cc111d640 10 client.4318.objecter close_session for osd.4 2024-01-21T04:33:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.139+0000 7f8cc111d640 1 -- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] conn(0x7f8ca8072c90 msgr2=0x7f8ca80750b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.139+0000 7f8cc111d640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] conn(0x7f8ca8072c90 0x7f8ca80750b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cc111d640 20 client.4318.objecter put_session s=0x7f8ca80725c0 osd=4 1 2024-01-21T04:33:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cc111d640 20 client.4318.objecter shutdown clearing up homeless session... 2024-01-21T04:33:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cc111d640 10 client.4318.objecter successfully canceled tick 2024-01-21T04:33:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cc111d640 1 -- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8ca806c8b0 msgr2=0x7f8ca806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cc111d640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8ca806c8b0 0x7f8ca806ed70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4036000 tx=0x7f8cb402e000 comp rx=0 tx=0).stop 2024-01-21T04:33:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cc111d640 10 monclient: shutdown 2024-01-21T04:33:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cc111d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cc111d640 1 -- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cbc0b2020 msgr2=0x7f8cbc1d81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cc111d640 1 --2- 172.21.15.69:0/3774713728 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cbc0b2020 0x7f8cbc1d81f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000c390 tx=0x7f8cb000c850 comp rx=0 tx=0).stop 2024-01-21T04:33:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cbb59e640 1 -- 172.21.15.69:0/3774713728 reap_dead start 2024-01-21T04:33:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cc111d640 1 -- 172.21.15.69:0/3774713728 shutdown_connections 2024-01-21T04:33:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f8cc111d640 1 -- 172.21.15.69:0/3774713728 >> 172.21.15.69:0/3774713728 conn(0x7f8cbc066cd0 msgr2=0x7f8cbc1425b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f826a7fc640 1 -- 172.21.15.69:0/2515081389 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8264041030 con 0x7f826c0c2e20 2024-01-21T04:33:50.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.140+0000 7f826a7fc640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f825806c8b0 0x7f825806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7f8279d25640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f825806c8b0 0x7f825806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7f8279d25640 10 monclient: get_auth_request con 0x7f825806c8b0 auth_method 0 2024-01-21T04:33:50.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7f8cc111d640 1 -- 172.21.15.69:0/3774713728 shutdown_connections 2024-01-21T04:33:50.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7f8cc111d640 1 -- 172.21.15.69:0/3774713728 wait complete. 2024-01-21T04:33:50.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7f8cc111d640 1 librados: shutdown 2024-01-21T04:33:50.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7f8279d25640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f825806c8b0 0x7f825806ed70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8270001560 tx=0x7f8270007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7f826a7fc640 10 client.4280.objecter ms_handle_connect 0x7f825806c8b0 2024-01-21T04:33:50.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7fdc3ffff640 1 -- 172.21.15.69:0/1399184110 <== osd.1 v2:172.21.15.69:6808/3868480563 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdc480058b0 con 0x7fdc44072c90 2024-01-21T04:33:50.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7fdc3ffff640 10 client.4302.objecter ms_dispatch 0x7fdc58000b90 command_reply(tid 2: 0 ) v1 2024-01-21T04:33:50.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7fdc3ffff640 10 client.4302.objecter _finish_command 2 = system:0 2024-01-21T04:33:50.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7fdc3ffff640 20 client.4302.objecter put_session s=0x7fdc440725c0 osd=1 4 2024-01-21T04:33:50.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.141+0000 7fdc3ffff640 15 client.4302.objecter _session_command_op_remove 1 2 2024-01-21T04:33:50.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 10 librados: watch_flush enter 2024-01-21T04:33:50.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 10 librados: watch_flush exit 2024-01-21T04:33:50.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 10 client.4302.objecter close_session for osd.1 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1399184110 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x7fdc44072c90 msgr2=0x7fdc440750b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x7fdc44072c90 0x7fdc440750b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 20 client.4302.objecter put_session s=0x7fdc440725c0 osd=1 1 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 20 client.4302.objecter shutdown clearing up homeless session... 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 10 client.4302.objecter successfully canceled tick 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1399184110 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdc4406c8b0 msgr2=0x7fdc4406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdc4406c8b0 0x7fdc4406ed70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdc50036000 tx=0x7fdc5002e000 comp rx=0 tx=0).stop 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 10 monclient: shutdown 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1399184110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5806ec10 msgr2=0x7fdc580acb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 1 --2- 172.21.15.69:0/1399184110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc5806ec10 0x7fdc580acb90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00c430 tx=0x7fdc4c00c8f0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc57fff640 1 -- 172.21.15.69:0/1399184110 reap_dead start 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1399184110 shutdown_connections 2024-01-21T04:33:50.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.142+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1399184110 >> 172.21.15.69:0/1399184110 conn(0x7fdc580b3960 msgr2=0x7fdc58073df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.143+0000 7f44ea7fc640 1 -- 172.21.15.69:0/1822962835 <== osd.0 v2:172.21.15.69:6824/335500148 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f44f00012b0 con 0x7f44d8072c40 2024-01-21T04:33:50.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.143+0000 7f44ea7fc640 10 client.4321.objecter ms_dispatch 0x7f44f4000b90 command_reply(tid 2: 0 ) v1 2024-01-21T04:33:50.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.143+0000 7f44ea7fc640 10 client.4321.objecter _finish_command 2 = system:0 2024-01-21T04:33:50.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.143+0000 7f44ea7fc640 20 client.4321.objecter put_session s=0x7f44d8072570 osd=0 4 2024-01-21T04:33:50.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.143+0000 7f44ea7fc640 15 client.4321.objecter _session_command_op_remove 0 2 2024-01-21T04:33:50.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.143+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1399184110 shutdown_connections 2024-01-21T04:33:50.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.143+0000 7fdc5cf6c640 1 -- 172.21.15.69:0/1399184110 wait complete. 2024-01-21T04:33:50.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7fdc5cf6c640 1 librados: shutdown 2024-01-21T04:33:50.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7f44fad3c640 10 librados: watch_flush enter 2024-01-21T04:33:50.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7f44fad3c640 10 librados: watch_flush exit 2024-01-21T04:33:50.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.143+0000 7f0219ffb640 1 -- 172.21.15.69:0/417121987 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f021404d3f0 con 0x7f02240a9d10 2024-01-21T04:33:50.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7f0219ffb640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f020806c860 0x7f020806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7f44fad3c640 10 client.4321.objecter close_session for osd.0 2024-01-21T04:33:50.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7f44fad3c640 1 -- 172.21.15.69:0/1822962835 >> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] conn(0x7f44d8072c40 msgr2=0x7f44d8075060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7f44fad3c640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] conn(0x7f44d8072c40 0x7f44d8075060 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7f44fad3c640 20 client.4321.objecter put_session s=0x7f44d8072570 osd=0 1 2024-01-21T04:33:50.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7f44fad3c640 20 client.4321.objecter shutdown clearing up homeless session... 2024-01-21T04:33:50.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7f44fad3c640 10 client.4321.objecter successfully canceled tick 2024-01-21T04:33:50.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7f0229152640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f020806c860 0x7f020806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.144+0000 7f0229152640 10 monclient: get_auth_request con 0x7f020806c860 auth_method 0 2024-01-21T04:33:50.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.145+0000 7f44fad3c640 1 -- 172.21.15.69:0/1822962835 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f44d806c860 msgr2=0x7f44d806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.145+0000 7f44fad3c640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f44d806c860 0x7f44d806ed20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f44f406e870 tx=0x7f44e4031000 comp rx=0 tx=0).stop 2024-01-21T04:33:50.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.145+0000 7f44fad3c640 10 monclient: shutdown 2024-01-21T04:33:50.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.145+0000 7f44fad3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.145+0000 7f44fad3c640 1 -- 172.21.15.69:0/1822962835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44f413b760 msgr2=0x7f44f4136850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.145+0000 7f44fad3c640 1 --2- 172.21.15.69:0/1822962835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f44f413b760 0x7f44f4136850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f44ec00b9e0 tx=0x7f44ec00bea0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.145+0000 7f0229152640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f020806c860 0x7f020806ed20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0220009710 tx=0x7f0220006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.145+0000 7f0219ffb640 10 client.4330.objecter ms_handle_connect 0x7f020806c860 2024-01-21T04:33:50.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.145+0000 7f44fa53b640 1 -- 172.21.15.69:0/1822962835 reap_dead start 2024-01-21T04:33:50.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.145+0000 7f44fad3c640 1 -- 172.21.15.69:0/1822962835 shutdown_connections 2024-01-21T04:33:50.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.145+0000 7f44fad3c640 1 -- 172.21.15.69:0/1822962835 >> 172.21.15.69:0/1822962835 conn(0x7f44f40b0040 msgr2=0x7f44f40ae150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.146+0000 7f826a7fc640 1 -- 172.21.15.69:0/2515081389 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f826407afb0 con 0x7f826c0c2e20 2024-01-21T04:33:50.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.146+0000 7f826a7fc640 10 client.4280.objecter ms_dispatch 0x7f8274000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T04:33:50.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.146+0000 7f826a7fc640 3 client.4280.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T04:33:50.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.146+0000 7f826a7fc640 3 client.4280.objecter handle_osd_map decoding full epoch 25 2024-01-21T04:33:50.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 10 client.4280.objecter checking command 1 2024-01-21T04:33:50.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] conn(0x7f8258072c90 0x7f82580750b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 20 client.4280.objecter _get_session s=0x7f82580725c0 osd=2 3 2024-01-21T04:33:50.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 20 client.4280.objecter put_session s=0x7f82580725c0 osd=2 3 2024-01-21T04:33:50.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 15 client.4280.objecter _session_command_op_remove -1 1 2024-01-21T04:33:50.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 20 client.4280.objecter _get_session s=0x7f82580725c0 osd=2 3 2024-01-21T04:33:50.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 20 client.4280.objecter get_session s=0x7f82580725c0 osd=2 3 2024-01-21T04:33:50.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 15 client.4280.objecter _session_command_op_assign 2 1 2024-01-21T04:33:50.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 20 client.4280.objecter put_session s=0x7f82580725c0 osd=2 4 2024-01-21T04:33:50.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 10 client.4280.objecter _send_command 1 2024-01-21T04:33:50.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 1 -- 172.21.15.69:0/2515081389 --> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8258075760 con 0x7f8258072c90 2024-01-21T04:33:50.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f826a7fc640 20 client.4280.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.147+0000 7f827a526640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] conn(0x7f8258072c90 0x7f82580750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.148+0000 7f827a526640 10 monclient: get_auth_request con 0x7f8258072c90 auth_method 0 2024-01-21T04:33:50.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.151+0000 7f3b3bfff640 1 -- 172.21.15.69:0/776366037 <== osd.6 v2:172.21.15.144:6812/479183693 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3b40075710 con 0x7f3b40072c40 2024-01-21T04:33:50.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.151+0000 7f3b3bfff640 10 client.4286.objecter ms_dispatch 0x7f3b54000b90 command_reply(tid 1: 0 ) v1 2024-01-21T04:33:50.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.151+0000 7f3b3bfff640 10 client.4286.objecter _finish_command 1 = system:0 2024-01-21T04:33:50.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.151+0000 7f3b3bfff640 20 client.4286.objecter put_session s=0x7f3b40072570 osd=6 4 2024-01-21T04:33:50.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.151+0000 7f3b3bfff640 15 client.4286.objecter _session_command_op_remove 6 1 2024-01-21T04:33:50.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.152+0000 7f44fad3c640 1 -- 172.21.15.69:0/1822962835 shutdown_connections 2024-01-21T04:33:50.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.154+0000 7f826a7fc640 10 client.4280.objecter ms_handle_connect 0x7f8258072c90 2024-01-21T04:33:50.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.154+0000 7f827a526640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] conn(0x7f8258072c90 0x7f82580750b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 1 -- 172.21.15.69:0/417121987 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f021407a990 con 0x7f02240a9d10 2024-01-21T04:33:50.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 10 client.4330.objecter ms_dispatch 0x7f0224000b90 osd_map(25..25 src has 1..25) v4 2024-01-21T04:33:50.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 3 client.4330.objecter handle_osd_map got epochs [25,25] > 0 2024-01-21T04:33:50.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 3 client.4330.objecter handle_osd_map decoding full epoch 25 2024-01-21T04:33:50.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 10 client.4330.objecter checking command 1 2024-01-21T04:33:50.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] conn(0x7f0208072c40 0x7f0208075060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 20 client.4330.objecter _get_session s=0x7f0208072570 osd=7 3 2024-01-21T04:33:50.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 20 client.4330.objecter put_session s=0x7f0208072570 osd=7 3 2024-01-21T04:33:50.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 15 client.4330.objecter _session_command_op_remove -1 1 2024-01-21T04:33:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 20 client.4330.objecter _get_session s=0x7f0208072570 osd=7 3 2024-01-21T04:33:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 20 client.4330.objecter get_session s=0x7f0208072570 osd=7 3 2024-01-21T04:33:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 15 client.4330.objecter _session_command_op_assign 7 1 2024-01-21T04:33:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 20 client.4330.objecter put_session s=0x7f0208072570 osd=7 4 2024-01-21T04:33:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 10 client.4330.objecter _send_command 1 2024-01-21T04:33:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 1 -- 172.21.15.69:0/417121987 --> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0208075710 con 0x7f0208072c40 2024-01-21T04:33:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.153+0000 7f0219ffb640 20 client.4330.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.154+0000 7f0229953640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] conn(0x7f0208072c40 0x7f0208075060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.154+0000 7f0229953640 10 monclient: get_auth_request con 0x7f0208072c40 auth_method 0 2024-01-21T04:33:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.156+0000 7f44fad3c640 1 -- 172.21.15.69:0/1822962835 wait complete. 2024-01-21T04:33:50.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.156+0000 7f44fad3c640 1 librados: shutdown 2024-01-21T04:33:50.162 INFO:teuthology.orchestra.run.smithi069.stdout:85899345928 2024-01-21T04:33:50.163 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-21T04:33:50.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.162+0000 7f0229953640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] conn(0x7f0208072c40 0x7f0208075060 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.162+0000 7f0219ffb640 10 client.4330.objecter ms_handle_connect 0x7f0208072c40 2024-01-21T04:33:50.164 INFO:teuthology.orchestra.run.smithi069.stdout:64424509446 2024-01-21T04:33:50.164 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-21T04:33:50.172 INFO:teuthology.orchestra.run.smithi069.stdout:64424509446 2024-01-21T04:33:50.173 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-21T04:33:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.176+0000 7f3b5a105640 10 client.4286.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T04:33:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.176+0000 7f3b5a105640 15 client.4286.objecter _session_command_op_assign -1 2 2024-01-21T04:33:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.176+0000 7f3b5a105640 20 client.4286.objecter _get_session s=0x7f3b40072570 osd=6 3 2024-01-21T04:33:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.176+0000 7f3b5a105640 20 client.4286.objecter put_session s=0x7f3b40072570 osd=6 3 2024-01-21T04:33:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.176+0000 7f3b5a105640 20 client.4286.objecter _get_session s=0x7f3b40072570 osd=6 3 2024-01-21T04:33:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.176+0000 7f3b5a105640 15 client.4286.objecter _session_command_op_remove -1 2 2024-01-21T04:33:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.176+0000 7f3b5a105640 20 client.4286.objecter get_session s=0x7f3b40072570 osd=6 3 2024-01-21T04:33:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.176+0000 7f3b5a105640 15 client.4286.objecter _session_command_op_assign 6 2 2024-01-21T04:33:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.176+0000 7f3b5a105640 20 client.4286.objecter put_session s=0x7f3b40072570 osd=6 4 2024-01-21T04:33:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.177+0000 7f3b5a105640 10 client.4286.objecter _send_command 2 2024-01-21T04:33:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.177+0000 7f3b5a105640 1 -- 172.21.15.69:0/776366037 --> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3b34001240 con 0x7f3b40072c40 2024-01-21T04:33:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.183+0000 7f3b3bfff640 1 -- 172.21.15.69:0/776366037 <== osd.6 v2:172.21.15.144:6812/479183693 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3b34001240 con 0x7f3b40072c40 2024-01-21T04:33:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.183+0000 7f3b3bfff640 10 client.4286.objecter ms_dispatch 0x7f3b54000b90 command_reply(tid 2: 0 ) v1 2024-01-21T04:33:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.183+0000 7f3b3bfff640 10 client.4286.objecter _finish_command 2 = system:0 2024-01-21T04:33:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.183+0000 7f3b3bfff640 20 client.4286.objecter put_session s=0x7f3b40072570 osd=6 4 2024-01-21T04:33:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.183+0000 7f3b3bfff640 15 client.4286.objecter _session_command_op_remove 6 2 2024-01-21T04:33:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 10 librados: watch_flush enter 2024-01-21T04:33:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 10 librados: watch_flush exit 2024-01-21T04:33:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 10 client.4286.objecter close_session for osd.6 2024-01-21T04:33:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 1 -- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] conn(0x7f3b40072c40 msgr2=0x7f3b40075060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] conn(0x7f3b40072c40 0x7f3b40075060 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 20 client.4286.objecter put_session s=0x7f3b40072570 osd=6 1 2024-01-21T04:33:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 20 client.4286.objecter shutdown clearing up homeless session... 2024-01-21T04:33:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 10 client.4286.objecter successfully canceled tick 2024-01-21T04:33:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 1 -- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f3b4006c860 msgr2=0x7f3b4006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f3b4006c860 0x7f3b4006ed20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3b44001560 tx=0x7f3b44007990 comp rx=0 tx=0).stop 2024-01-21T04:33:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 10 monclient: shutdown 2024-01-21T04:33:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 20 monclient: shutdown canceling and discarding version request 1 2024-01-21T04:33:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 1 -- 172.21.15.69:0/776366037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b54143490 msgr2=0x7f3b5413f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 1 --2- 172.21.15.69:0/776366037 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f3b54143490 0x7f3b5413f5e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00b9a0 tx=0x7f3b4c00be60 comp rx=0 tx=0).stop 2024-01-21T04:33:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b53fff640 1 -- 172.21.15.69:0/776366037 reap_dead start 2024-01-21T04:33:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 1 -- 172.21.15.69:0/776366037 shutdown_connections 2024-01-21T04:33:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.184+0000 7f3b5a105640 1 -- 172.21.15.69:0/776366037 >> 172.21.15.69:0/776366037 conn(0x7f3b540b4c60 msgr2=0x7f3b540b2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.185+0000 7f3b5a105640 1 -- 172.21.15.69:0/776366037 shutdown_connections 2024-01-21T04:33:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.185+0000 7f3b5a105640 1 -- 172.21.15.69:0/776366037 wait complete. 2024-01-21T04:33:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.185+0000 7f3b5a105640 1 librados: shutdown 2024-01-21T04:33:50.207 INFO:teuthology.orchestra.run.smithi069.stdout:85899345928 2024-01-21T04:33:50.207 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-21T04:33:50.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.209+0000 7f0219ffb640 1 -- 172.21.15.69:0/417121987 <== osd.7 v2:172.21.15.144:6810/14156089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0208075710 con 0x7f0208072c40 2024-01-21T04:33:50.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.209+0000 7f0219ffb640 10 client.4330.objecter ms_dispatch 0x7f0224000b90 command_reply(tid 1: 0 ) v1 2024-01-21T04:33:50.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.209+0000 7f0219ffb640 10 client.4330.objecter _finish_command 1 = system:0 2024-01-21T04:33:50.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.209+0000 7f0219ffb640 20 client.4330.objecter put_session s=0x7f0208072570 osd=7 4 2024-01-21T04:33:50.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.209+0000 7f0219ffb640 15 client.4330.objecter _session_command_op_remove 7 1 2024-01-21T04:33:50.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.231+0000 7f826a7fc640 1 -- 172.21.15.69:0/2515081389 <== osd.2 v2:172.21.15.69:6816/3954012353 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8258075760 con 0x7f8258072c90 2024-01-21T04:33:50.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.231+0000 7f826a7fc640 10 client.4280.objecter ms_dispatch 0x7f8274000b90 command_reply(tid 1: 0 ) v1 2024-01-21T04:33:50.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.232+0000 7f826a7fc640 10 client.4280.objecter _finish_command 1 = system:0 2024-01-21T04:33:50.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.232+0000 7f826a7fc640 20 client.4280.objecter put_session s=0x7f82580725c0 osd=2 4 2024-01-21T04:33:50.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.232+0000 7f826a7fc640 15 client.4280.objecter _session_command_op_remove 2 1 2024-01-21T04:33:50.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.249+0000 7ff31effd640 1 -- 172.21.15.69:0/4075847385 <== osd.3 v2:172.21.15.69:6800/1086763384 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff310075710 con 0x7ff310072c40 2024-01-21T04:33:50.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.249+0000 7ff31effd640 10 client.4277.objecter ms_dispatch 0x7ff330000b90 command_reply(tid 1: 0 ) v1 2024-01-21T04:33:50.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.249+0000 7ff31effd640 10 client.4277.objecter _finish_command 1 = system:0 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.249+0000 7ff31effd640 20 client.4277.objecter put_session s=0x7ff310072570 osd=3 4 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.249+0000 7ff31effd640 15 client.4277.objecter _session_command_op_remove 3 1 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.251+0000 7f022b3b6640 10 client.4330.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.251+0000 7f022b3b6640 15 client.4330.objecter _session_command_op_assign -1 2 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.251+0000 7f022b3b6640 20 client.4330.objecter _get_session s=0x7f0208072570 osd=7 3 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.251+0000 7f022b3b6640 20 client.4330.objecter put_session s=0x7f0208072570 osd=7 3 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.251+0000 7f022b3b6640 20 client.4330.objecter _get_session s=0x7f0208072570 osd=7 3 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.251+0000 7f022b3b6640 15 client.4330.objecter _session_command_op_remove -1 2 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.251+0000 7f022b3b6640 20 client.4330.objecter get_session s=0x7f0208072570 osd=7 3 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.251+0000 7f022b3b6640 15 client.4330.objecter _session_command_op_assign 7 2 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.251+0000 7f022b3b6640 20 client.4330.objecter put_session s=0x7f0208072570 osd=7 4 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.251+0000 7f022b3b6640 10 client.4330.objecter _send_command 2 2024-01-21T04:33:50.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.251+0000 7f022b3b6640 1 -- 172.21.15.69:0/417121987 --> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0224142ef0 con 0x7f0208072c40 2024-01-21T04:33:50.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.257+0000 7f0219ffb640 1 -- 172.21.15.69:0/417121987 <== osd.7 v2:172.21.15.144:6810/14156089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0224142ef0 con 0x7f0208072c40 2024-01-21T04:33:50.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.257+0000 7f0219ffb640 10 client.4330.objecter ms_dispatch 0x7f0224000b90 command_reply(tid 2: 0 ) v1 2024-01-21T04:33:50.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.257+0000 7f0219ffb640 10 client.4330.objecter _finish_command 2 = system:0 2024-01-21T04:33:50.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.257+0000 7f0219ffb640 20 client.4330.objecter put_session s=0x7f0208072570 osd=7 4 2024-01-21T04:33:50.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.257+0000 7f0219ffb640 15 client.4330.objecter _session_command_op_remove 7 2 2024-01-21T04:33:50.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.262+0000 7f022b3b6640 10 librados: watch_flush enter 2024-01-21T04:33:50.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 10 librados: watch_flush exit 2024-01-21T04:33:50.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 10 client.4330.objecter close_session for osd.7 2024-01-21T04:33:50.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 1 -- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] conn(0x7f0208072c40 msgr2=0x7f0208075060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] conn(0x7f0208072c40 0x7f0208075060 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 20 client.4330.objecter put_session s=0x7f0208072570 osd=7 1 2024-01-21T04:33:50.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 20 client.4330.objecter shutdown clearing up homeless session... 2024-01-21T04:33:50.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 10 client.4330.objecter successfully canceled tick 2024-01-21T04:33:50.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 1 -- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f020806c860 msgr2=0x7f020806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f020806c860 0x7f020806ed20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0220009710 tx=0x7f0220006c90 comp rx=0 tx=0).stop 2024-01-21T04:33:50.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 10 monclient: shutdown 2024-01-21T04:33:50.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 20 monclient: shutdown canceling and discarding version request 1 2024-01-21T04:33:50.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 1 -- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02240a9d10 msgr2=0x7f0224142290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f022b3b6640 1 --2- 172.21.15.69:0/417121987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02240a9d10 0x7f0224142290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0214000c00 tx=0x7f02140046c0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.263+0000 7f0229953640 1 -- 172.21.15.69:0/417121987 reap_dead start 2024-01-21T04:33:50.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.264+0000 7f022b3b6640 1 -- 172.21.15.69:0/417121987 shutdown_connections 2024-01-21T04:33:50.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.264+0000 7f022b3b6640 1 -- 172.21.15.69:0/417121987 >> 172.21.15.69:0/417121987 conn(0x7f0224066fb0 msgr2=0x7f0224133f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f022b3b6640 1 -- 172.21.15.69:0/417121987 shutdown_connections 2024-01-21T04:33:50.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f022b3b6640 1 -- 172.21.15.69:0/417121987 wait complete. 2024-01-21T04:33:50.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f022b3b6640 1 librados: shutdown 2024-01-21T04:33:50.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f827ad27640 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T04:33:50.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.270+0000 7ff31effd640 1 -- 172.21.15.69:0/4075847385 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff324076e20 con 0x7ff3280322a0 2024-01-21T04:33:50.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.270+0000 7ff31effd640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T04:33:50.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f827ad27640 15 client.4280.objecter _session_command_op_assign -1 2 2024-01-21T04:33:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f827ad27640 20 client.4280.objecter _get_session s=0x7f82580725c0 osd=2 3 2024-01-21T04:33:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f827ad27640 20 client.4280.objecter put_session s=0x7f82580725c0 osd=2 3 2024-01-21T04:33:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f827ad27640 20 client.4280.objecter _get_session s=0x7f82580725c0 osd=2 3 2024-01-21T04:33:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f827ad27640 15 client.4280.objecter _session_command_op_remove -1 2 2024-01-21T04:33:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f827ad27640 20 client.4280.objecter get_session s=0x7f82580725c0 osd=2 3 2024-01-21T04:33:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f827ad27640 15 client.4280.objecter _session_command_op_assign 2 2 2024-01-21T04:33:50.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f827ad27640 20 client.4280.objecter put_session s=0x7f82580725c0 osd=2 4 2024-01-21T04:33:50.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f827ad27640 10 client.4280.objecter _send_command 2 2024-01-21T04:33:50.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.267+0000 7f827ad27640 1 -- 172.21.15.69:0/2515081389 --> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f826c0b9c20 con 0x7f8258072c90 2024-01-21T04:33:50.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.275+0000 7f826a7fc640 1 -- 172.21.15.69:0/2515081389 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8264076bf0 con 0x7f826c0c2e20 2024-01-21T04:33:50.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.275+0000 7f826a7fc640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T04:33:50.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.288+0000 7f826a7fc640 1 -- 172.21.15.69:0/2515081389 <== osd.2 v2:172.21.15.69:6816/3954012353 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f826c0b9c20 con 0x7f8258072c90 2024-01-21T04:33:50.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.288+0000 7f826a7fc640 10 client.4280.objecter ms_dispatch 0x7f8274000b90 command_reply(tid 2: 0 ) v1 2024-01-21T04:33:50.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.288+0000 7f826a7fc640 10 client.4280.objecter _finish_command 2 = system:0 2024-01-21T04:33:50.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.288+0000 7f826a7fc640 20 client.4280.objecter put_session s=0x7f82580725c0 osd=2 4 2024-01-21T04:33:50.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.288+0000 7f826a7fc640 15 client.4280.objecter _session_command_op_remove 2 2 2024-01-21T04:33:50.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.288+0000 7f8257fff640 10 librados: watch_flush enter 2024-01-21T04:33:50.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.288+0000 7f8257fff640 10 librados: watch_flush exit 2024-01-21T04:33:50.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.288+0000 7f8257fff640 10 client.4280.objecter close_session for osd.2 2024-01-21T04:33:50.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 1 -- 172.21.15.69:0/2515081389 >> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] conn(0x7f8258072c90 msgr2=0x7f82580750b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] conn(0x7f8258072c90 0x7f82580750b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 20 client.4280.objecter put_session s=0x7f82580725c0 osd=2 1 2024-01-21T04:33:50.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 20 client.4280.objecter shutdown clearing up homeless session... 2024-01-21T04:33:50.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 10 client.4280.objecter successfully canceled tick 2024-01-21T04:33:50.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 1 -- 172.21.15.69:0/2515081389 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f825806c8b0 msgr2=0x7f825806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f825806c8b0 0x7f825806ed70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8270001560 tx=0x7f8270007990 comp rx=0 tx=0).stop 2024-01-21T04:33:50.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 10 monclient: shutdown 2024-01-21T04:33:50.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 1 -- 172.21.15.69:0/2515081389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 msgr2=0x7f826c0c51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 1 --2- 172.21.15.69:0/2515081389 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f826c0c2e20 0x7f826c0c51c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8264004000 tx=0x7f82640046c0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f827a526640 1 -- 172.21.15.69:0/2515081389 reap_dead start 2024-01-21T04:33:50.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 1 -- 172.21.15.69:0/2515081389 shutdown_connections 2024-01-21T04:33:50.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7f8257fff640 1 -- 172.21.15.69:0/2515081389 >> 172.21.15.69:0/2515081389 conn(0x7f826c006510 msgr2=0x7f826c02cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.290+0000 7f8257fff640 1 -- 172.21.15.69:0/2515081389 shutdown_connections 2024-01-21T04:33:50.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.290+0000 7f8257fff640 1 -- 172.21.15.69:0/2515081389 wait complete. 2024-01-21T04:33:50.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.291+0000 7f8257fff640 1 librados: shutdown 2024-01-21T04:33:50.293 INFO:teuthology.orchestra.run.smithi069.stdout:64424509448 2024-01-21T04:33:50.294 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-21T04:33:50.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7ff334d0f640 10 client.4277.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-01-21T04:33:50.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7ff334d0f640 15 client.4277.objecter _session_command_op_assign -1 2 2024-01-21T04:33:50.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7ff334d0f640 20 client.4277.objecter _get_session s=0x7ff310072570 osd=3 3 2024-01-21T04:33:50.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7ff334d0f640 20 client.4277.objecter put_session s=0x7ff310072570 osd=3 3 2024-01-21T04:33:50.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7ff334d0f640 20 client.4277.objecter _get_session s=0x7ff310072570 osd=3 3 2024-01-21T04:33:50.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7ff334d0f640 15 client.4277.objecter _session_command_op_remove -1 2 2024-01-21T04:33:50.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7ff334d0f640 20 client.4277.objecter get_session s=0x7ff310072570 osd=3 3 2024-01-21T04:33:50.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7ff334d0f640 15 client.4277.objecter _session_command_op_assign 3 2 2024-01-21T04:33:50.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7ff334d0f640 20 client.4277.objecter put_session s=0x7ff310072570 osd=3 4 2024-01-21T04:33:50.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7ff334d0f640 10 client.4277.objecter _send_command 2 2024-01-21T04:33:50.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.289+0000 7ff334d0f640 1 -- 172.21.15.69:0/4075847385 --> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff32802b9f0 con 0x7ff310072c40 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.300+0000 7fe3f2e07640 10 monclient: get_monmap_and_config 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.300+0000 7fe3f2e07640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.300+0000 7fe3f2e07640 10 monclient: monmap: 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:50.301915+0000 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:50.301915+0000 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:50.302 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:50.303 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:50.303 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:50.303 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.301+0000 7fe3f2e07640 1 Processor -- start 2024-01-21T04:33:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 1 -- start start 2024-01-21T04:33:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 10 monclient: init 2024-01-21T04:33:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-01-21T04:33:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe3ec0a8b20 0x7fe3ec145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 10 monclient(hunting): picked mon.noname-a con 0x7fe3ec0a8b20 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3ec0a9d10 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe3ec0b2310 0x7fe3ec13f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 10 monclient(hunting): picked mon.noname-c con 0x7fe3ec0b2310 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ec1456f0 con 0x7fe3ec0a9d10 2024-01-21T04:33:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ec0a98c0 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ec1460c0 con 0x7fe3ec0b2310 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.302+0000 7fe3f2e07640 10 monclient(hunting): authenticate will time out at 1388.644195s 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.303+0000 7fe3f0ba3640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe3ec0a8b20 0x7fe3ec145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.303+0000 7fe3f13a4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe3ec0b2310 0x7fe3ec13f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f13a4640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe3ec0b2310 0x7fe3ec13f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59746/0 (socket says 172.21.15.69:59746) 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f13a4640 1 -- 172.21.15.69:0/1644673654 learned_addr learned my addr 172.21.15.69:0/1644673654 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f13a4640 10 monclient(hunting): get_auth_request con 0x7fe3ec0b2310 auth_method 0 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f13a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f13a4640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f13a4640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f0ba3640 10 monclient(hunting): get_auth_request con 0x7fe3ec0a8b20 auth_method 0 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f0ba3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f0ba3640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f0ba3640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f0ba3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f0ba3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.304+0000 7fe3f0ba3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.305+0000 7fe3f13a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.305+0000 7fe3f13a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.305+0000 7fe3f13a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.307+0000 7fe3f0ba3640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.307+0000 7fe3f0ba3640 1 -- 172.21.15.69:0/1644673654 >> v2:172.21.15.144:3300/0 conn(0x7fe3ec0b2310 msgr2=0x7fe3ec13f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.307+0000 7fe3f0ba3640 1 --2- 172.21.15.69:0/1644673654 >> v2:172.21.15.144:3300/0 conn(0x7fe3ec0b2310 0x7fe3ec13f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.307+0000 7fe3f0ba3640 1 -- 172.21.15.69:0/1644673654 >> v1:172.21.15.144:6789/0 conn(0x7fe3ec0a9d10 legacy=0x7fe3ec13f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.307+0000 7fe3f0ba3640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.307+0000 7fe3f0ba3640 1 monclient: found mon.noname-a 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.307+0000 7fe3f0ba3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.307+0000 7fe3f0ba3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.307+0000 7fe3f0ba3640 1 -- 172.21.15.69:0/1644673654 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec146230 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.311 INFO:teuthology.orchestra.run.smithi069.stdout:64424509448 2024-01-21T04:33:50.311 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-21T04:33:50.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.312+0000 7fe3f0ba3640 1 --2- 172.21.15.69:0/1644673654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3ec0a8b20 0x7fe3ec145b80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec13fcd0 tx=0x7fe3e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3670c5a37360d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.316+0000 7fe3eb7fe640 1 -- 172.21.15.69:0/1644673654 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0031870 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.316+0000 7fe3eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.316+0000 7fe3eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.316+0000 7fe3eb7fe640 10 monclient: dump: 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.316+0000 7fe3eb7fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.316+0000 7fe3eb7fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.316+0000 7fe3eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.316+0000 7fe3f2e07640 5 monclient: authenticate success, global_id 4305 2024-01-21T04:33:50.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.316+0000 7fe3f2e07640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.320+0000 7fe3eb7fe640 1 -- 172.21.15.69:0/1644673654 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e00319d0 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.320+0000 7fe3eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.320+0000 7fe3eb7fe640 1 -- 172.21.15.69:0/1644673654 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0031da0 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.320+0000 7fe3eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.320+0000 7fe3eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.320+0000 7fe3eb7fe640 10 monclient: dump: 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.321 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.322 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.322 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.322 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.320+0000 7fe3f2e07640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:50.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.320+0000 7fe3f2e07640 4 set_mon_vals no callback set 2024-01-21T04:33:50.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.321+0000 7fe3f2e07640 10 monclient: shutdown 2024-01-21T04:33:50.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.321+0000 7fe3f2e07640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.321+0000 7fe3f2e07640 1 -- 172.21.15.69:0/1644673654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3ec0a8b20 msgr2=0x7fe3ec145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.321+0000 7fe3f2e07640 1 --2- 172.21.15.69:0/1644673654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3ec0a8b20 0x7fe3ec145b80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec13fcd0 tx=0x7fe3e002b910 comp rx=0 tx=0).stop 2024-01-21T04:33:50.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.323+0000 7fe3f2e07640 1 -- 172.21.15.69:0/1644673654 shutdown_connections 2024-01-21T04:33:50.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.323+0000 7fe3f2e07640 1 --2- 172.21.15.69:0/1644673654 >> v2:172.21.15.144:3300/0 conn(0x7fe3ec0b2310 0x7fe3ec13f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.323+0000 7fe3f2e07640 1 --2- 172.21.15.69:0/1644673654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3ec0a8b20 0x7fe3ec145b80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.323+0000 7fe3f2e07640 1 -- 172.21.15.69:0/1644673654 >> 172.21.15.69:0/1644673654 conn(0x7fe3ec066fb0 msgr2=0x7fe3ec0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.324+0000 7fe3f2e07640 1 -- 172.21.15.69:0/1644673654 shutdown_connections 2024-01-21T04:33:50.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.324+0000 7fe3f2e07640 1 -- 172.21.15.69:0/1644673654 wait complete. 2024-01-21T04:33:50.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.325+0000 7fe3f2e07640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.325+0000 7fe3f2e07640 10 monclient: monmap: 2024-01-21T04:33:50.327 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.327 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:50.327 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:50.327 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:50.327 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.328 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.328 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:50.328 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:50.328 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:50.328 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.325+0000 7fe3f2e07640 1 librados: starting msgr at 2024-01-21T04:33:50.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.325+0000 7fe3f2e07640 1 librados: starting objecter 2024-01-21T04:33:50.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.325+0000 7fe3f2e07640 1 Processor -- start 2024-01-21T04:33:50.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.326+0000 7fe3f2e07640 1 -- start start 2024-01-21T04:33:50.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.326+0000 7fe3f2e07640 1 librados: setting wanted keys 2024-01-21T04:33:50.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.326+0000 7fe3f2e07640 1 librados: calling monclient init 2024-01-21T04:33:50.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.326+0000 7fe3f2e07640 10 monclient: init 2024-01-21T04:33:50.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.327+0000 7fe3f2e07640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.327+0000 7fe3f2e07640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T04:33:50.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.327+0000 7fe3f2e07640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3ec0a8b20 0x7fe3ec144df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.327+0000 7fe3f2e07640 10 monclient(hunting): picked mon.noname-c con 0x7fe3ec0a8b20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.327+0000 7fe3f2e07640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3ec0a9d10 0x7fe3ec142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.330+0000 7fe3f0ba3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3ec0a8b20 0x7fe3ec144df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.330+0000 7fe3f0ba3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3ec0a8b20 0x7fe3ec144df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46892/0 (socket says 172.21.15.69:46892) 2024-01-21T04:33:50.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.330+0000 7fe3f0ba3640 1 -- 172.21.15.69:0/3704714150 learned_addr learned my addr 172.21.15.69:0/3704714150 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.330+0000 7fe3f2e07640 10 monclient(hunting): picked mon.noname-a con 0x7fe3ec0a9d10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.330+0000 7fe3f2e07640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3ec0b2310 0x7fe3ec1e7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.330+0000 7fe3f2e07640 10 monclient(hunting): picked mon.noname-b con 0x7fe3ec0b2310 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f2e07640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1320e0 con 0x7fe3ec0a9d10 2024-01-21T04:33:50.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f2e07640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec130630 con 0x7fe3ec0b2310 2024-01-21T04:33:50.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f2e07640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec132310 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f2e07640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f2e07640 10 monclient(hunting): authenticate will time out at 1388.672382s 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f0ba3640 10 monclient(hunting): get_auth_request con 0x7fe3ec0a8b20 auth_method 0 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f0ba3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f0ba3640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.331+0000 7fe3f0ba3640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.332+0000 7fe3f13a4640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3ec0b2310 0x7fe3ec1e7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.332+0000 7fe3f13a4640 10 monclient(hunting): get_auth_request con 0x7fe3ec0b2310 auth_method 0 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.332+0000 7fe3f13a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.332+0000 7fe3f13a4640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.332+0000 7fe3f13a4640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.332+0000 7fe3ebfff640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3ec0a9d10 0x7fe3ec142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.332+0000 7fe3ebfff640 10 monclient(hunting): get_auth_request con 0x7fe3ec0a9d10 auth_method 0 2024-01-21T04:33:50.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.332+0000 7fe3ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.332+0000 7fe3ebfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.332+0000 7fe3ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.333+0000 7fe3f0ba3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.333+0000 7fe3f0ba3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.333+0000 7fe3f0ba3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.333+0000 7fe3f13a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.333+0000 7fe3f13a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.333+0000 7fe3f13a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.334+0000 7fe3ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.334+0000 7fe3ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.334+0000 7fe3ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.340+0000 7fe3f0ba3640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2024-01-21T04:33:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.340+0000 7fe3f0ba3640 1 -- 172.21.15.69:0/3704714150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3ec0b2310 msgr2=0x7fe3ec1e7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.340+0000 7fe3f0ba3640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3ec0b2310 0x7fe3ec1e7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.340+0000 7fe3f0ba3640 1 -- 172.21.15.69:0/3704714150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3ec0a9d10 msgr2=0x7fe3ec142bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.340+0000 7fe3f0ba3640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3ec0a9d10 0x7fe3ec142bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.340+0000 7fe3f0ba3640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.340+0000 7fe3f0ba3640 1 monclient: found mon.noname-c 2024-01-21T04:33:50.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.340+0000 7fe3f0ba3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.340+0000 7fe3f0ba3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.340+0000 7fe3f0ba3640 1 -- 172.21.15.69:0/3704714150 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e0009400 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.344+0000 7fe3f0ba3640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3ec0a8b20 0x7fe3ec144df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009890 tx=0x7fe3e00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.344+0000 7fe3e97fa640 10 client.?.objecter ms_handle_connect 0x7fe3ec0a8b20 2024-01-21T04:33:50.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.344+0000 7fe3e97fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3e97fa640 1 -- 172.21.15.69:0/3704714150 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e003bc70 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3e97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:50.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3e97fa640 10 monclient: dump: 2024-01-21T04:33:50.348 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3e97fa640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3e97fa640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3f2e07640 5 monclient: authenticate success, global_id 4289 2024-01-21T04:33:50.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3f2e07640 10 monclient: _renew_subs 2024-01-21T04:33:50.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3f2e07640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec1325f0 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3f2e07640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3f2e07640 10 monclient: _renew_subs 2024-01-21T04:33:50.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3f2e07640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec1e82c0 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.347+0000 7fe3f2e07640 1 librados: init done 2024-01-21T04:33:50.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.349+0000 7fe3f2e07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.349+0000 7fe3f2e07640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.349+0000 7fe3f2e07640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.349+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ec04e970 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.353+0000 7fe3e97fa640 1 -- 172.21.15.69:0/3704714150 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e003bdd0 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.353+0000 7fe3e97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.359+0000 7fe3ea7fc640 4 set_mon_vals no callback set 2024-01-21T04:33:50.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.361+0000 7fe3e97fa640 1 -- 172.21.15.69:0/3704714150 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e004e5f0 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.361+0000 7fe3e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.361+0000 7fe3e97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.361+0000 7fe3e97fa640 10 monclient: dump: 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.362 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.362+0000 7f59aa9c0640 10 monclient: get_monmap_and_config 2024-01-21T04:33:50.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.362+0000 7f59aa9c0640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.362+0000 7f59aa9c0640 10 monclient: monmap: 2024-01-21T04:33:50.363 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.363 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.363 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:50.363068+0000 2024-01-21T04:33:50.363 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:50.363068+0000 2024-01-21T04:33:50.363 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.363 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.363 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:50.364 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:50.364 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:50.364 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:50.364 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:50.364 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.363+0000 7f59aa9c0640 1 Processor -- start 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.363+0000 7f59aa9c0640 1 -- start start 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.363+0000 7f59aa9c0640 10 monclient: init 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f59a40b0f60 0x7f59a40b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 10 monclient(hunting): picked mon.noname-c con 0x7f59a40b0f60 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59a40b3890 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59a41469b0 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a41473b0 con 0x7f59a41469b0 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a406ea10 con 0x7f59a40b3890 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a40b1340 con 0x7f59a40b0f60 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.364+0000 7f59aa9c0640 10 monclient(hunting): authenticate will time out at 1388.705475s 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.365+0000 7f59a3fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f59a40b0f60 0x7f59a40b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.365+0000 7f59a8f5d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f59a41469b0 0x7f59a4150db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58906/0 (socket says 172.21.15.69:58906) 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.365+0000 7f59a8f5d640 1 -- 172.21.15.69:0/1197830105 learned_addr learned my addr 172.21.15.69:0/1197830105 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.368+0000 7f59a3fff640 10 monclient(hunting): get_auth_request con 0x7f59a40b0f60 auth_method 0 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.368+0000 7f59a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.368+0000 7f59a3fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.368+0000 7f59a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.369+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519077228 0 0) 0x7f59a41473b0 con 0x7f59a41469b0 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.369+0000 7f59a2ffd640 10 monclient(hunting): _negotiate 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.369+0000 7f59a2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.369+0000 7f59a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.369+0000 7f59a2ffd640 10 monclient(hunting): my global_id is 4311 2024-01-21T04:33:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.369+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f598c003520 con 0x7f59a41469b0 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4107071661 0 0) 0x7f598c003520 con 0x7f59a41469b0 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 >> v2:172.21.15.144:3300/0 conn(0x7f59a40b0f60 msgr2=0x7f59a40b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 1 --2- 172.21.15.69:0/1197830105 >> v2:172.21.15.144:3300/0 conn(0x7f59a40b0f60 0x7f59a40b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 >> v1:172.21.15.144:6789/0 conn(0x7f59a40b3890 legacy=0x7f59a40b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 1 monclient: found mon.noname-a-legacy 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.69:6789/0 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a406ecf0 con 0x7f59a41469b0 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59aa9c0640 5 monclient: authenticate success, global_id 4311 2024-01-21T04:33:50.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59aa9c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 125097436 0 0) 0x7f599c0019f0 con 0x7f59a41469b0 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 10 monclient: dump: 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] but i'm connected to v1:172.21.15.69:6789/0, reconnecting 2024-01-21T04:33:50.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 >> v1:172.21.15.69:6789/0 conn(0x7f59a41469b0 legacy=0x7f59a4150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.375+0000 7f59a2ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:33:50.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 1 --2- 172.21.15.69:0/1197830105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f598c0039c0 0x7f598c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 10 monclient(hunting): picked mon.a con 0x7f598c0039c0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 1 --2- 172.21.15.69:0/1197830105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f598c0047c0 0x7f598c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7fcee1ea3640 10 monclient: get_monmap_and_config 2024-01-21T04:33:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7fcee1ea3640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7fcee1ea3640 10 monclient: monmap: 2024-01-21T04:33:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:50.377400+0000 2024-01-21T04:33:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:50.377400+0000 2024-01-21T04:33:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:50.380 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:50.381 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:50.381 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 10 monclient(hunting): picked mon.b con 0x7f598c0047c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 1 --2- 172.21.15.69:0/1197830105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c005250 0x7f598c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 10 monclient(hunting): picked mon.c con 0x7f598c005250 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a40b1340 con 0x7f598c0039c0 2024-01-21T04:33:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c00c1a0 con 0x7f598c0047c0 2024-01-21T04:33:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c00c360 con 0x7f598c005250 2024-01-21T04:33:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.376+0000 7f59a2ffd640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.377+0000 7f59a2ffd640 10 monclient(hunting): _finish_auth 0 2024-01-21T04:33:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.377+0000 7f59aa9c0640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.377+0000 7fcee1ea3640 1 Processor -- start 2024-01-21T04:33:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.377+0000 7fcee1ea3640 1 -- start start 2024-01-21T04:33:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.377+0000 7fcee1ea3640 10 monclient: init 2024-01-21T04:33:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-01-21T04:33:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fced402b360 0x7fced40c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 10 monclient(hunting): picked mon.noname-c con 0x7fced402b360 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fced402b740 0x7fced40c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 10 monclient(hunting): picked mon.noname-b con 0x7fced402b740 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fced40c50e0 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced402d620 con 0x7fced40c50e0 2024-01-21T04:33:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced402d900 con 0x7fced402b360 2024-01-21T04:33:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced40c2400 con 0x7fced402b740 2024-01-21T04:33:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7fcee1ea3640 10 monclient(hunting): authenticate will time out at 1388.719897s 2024-01-21T04:33:50.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.378+0000 7f59a3fff640 1 --2- 172.21.15.69:0/1197830105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f598c0039c0 0x7f598c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.379+0000 7f59a37fe640 1 --2- 172.21.15.69:0/1197830105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f598c0047c0 0x7f598c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.379+0000 7f59a3fff640 10 monclient(hunting): get_auth_request con 0x7f598c0039c0 auth_method 0 2024-01-21T04:33:50.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.379+0000 7f59a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.379+0000 7f59a3fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.379+0000 7f59a3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T04:33:50.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.379+0000 7f59a37fe640 10 monclient(hunting): get_auth_request con 0x7f598c0047c0 auth_method 0 2024-01-21T04:33:50.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.379+0000 7f59a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.379+0000 7f59a37fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.379+0000 7f59a37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T04:33:50.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.380+0000 7f59a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.380+0000 7f59a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.380+0000 7f59a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T04:33:50.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.380+0000 7fcee0ea1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fced402b360 0x7fced40c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.380+0000 7fcee0ea1640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fced402b360 0x7fced40c1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59788/0 (socket says 172.21.15.69:59788) 2024-01-21T04:33:50.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.380+0000 7fcee0ea1640 1 -- 172.21.15.69:0/2414602065 learned_addr learned my addr 172.21.15.69:0/2414602065 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.380+0000 7fcee0ea1640 10 monclient(hunting): get_auth_request con 0x7fced402b360 auth_method 0 2024-01-21T04:33:50.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.380+0000 7fcee0ea1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.380+0000 7fcee0ea1640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.380+0000 7fcee0ea1640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.381+0000 7f59a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.381+0000 7f59a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.381+0000 7f59a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T04:33:50.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 10 monclient(hunting): handle_auth_done global_id 4311 payload 293 2024-01-21T04:33:50.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 1 -- 172.21.15.69:0/1197830105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c005250 msgr2=0x7f598c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 1 --2- 172.21.15.69:0/1197830105 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f598c005250 0x7f598c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 1 -- 172.21.15.69:0/1197830105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f598c0039c0 msgr2=0x7f598c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 1 --2- 172.21.15.69:0/1197830105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f598c0039c0 0x7f598c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 1 monclient: found mon.b 2024-01-21T04:33:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 1 -- 172.21.15.69:0/1197830105 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c003520 con 0x7f598c0047c0 2024-01-21T04:33:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.382+0000 7f59a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.383+0000 7f59a8f5d640 1 -- 172.21.15.69:0/1197830105 reap_dead start 2024-01-21T04:33:50.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.384+0000 7fcee0ea1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.384+0000 7fcee0ea1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.384+0000 7fcee0ea1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.385+0000 7f59a37fe640 1 --2- 172.21.15.69:0/1197830105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f598c0047c0 0x7f598c004c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f59a40b54d0 tx=0x7f598c0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1427ecb141a99623 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fcee0ea1640 10 monclient(hunting): handle_auth_done global_id 4345 payload 274 2024-01-21T04:33:50.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fcee0ea1640 1 -- 172.21.15.69:0/2414602065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fced402b740 msgr2=0x7fced40c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fcee0ea1640 1 --2- 172.21.15.69:0/2414602065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fced402b740 0x7fced40c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fcee0ea1640 1 -- 172.21.15.69:0/2414602065 >> v1:172.21.15.69:6789/0 conn(0x7fced40c50e0 legacy=0x7fced40c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fcee0ea1640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fcee0ea1640 1 monclient: found mon.noname-c 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fcee0ea1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fcee0ea1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fcee0ea1640 1 -- 172.21.15.69:0/2414602065 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced402c760 con 0x7fced402b360 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fa1418b4640 10 monclient: get_monmap_and_config 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fa1418b4640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.387+0000 7fa1418b4640 10 monclient: monmap: 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:50.388252+0000 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:50.388252+0000 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.390 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.391 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:50.391 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:50.391 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:50.391 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:50.391 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:50.391 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.389+0000 7fa1418b4640 1 Processor -- start 2024-01-21T04:33:50.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fe3e97fa640 1 -- 172.21.15.69:0/3704714150 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe3e00453f0 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fe3e97fa640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe3d006c890 0x7fe3d006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fe3ebfff640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe3d006c890 0x7fe3d006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.393+0000 7fcee0ea1640 1 --2- 172.21.15.69:0/2414602065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fced402b360 0x7fced40c1b50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcec800bb10 tx=0x7fcec802dfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2249035745d1229d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.393+0000 7fe3ebfff640 10 monclient: get_auth_request con 0x7fe3d006c890 auth_method 0 2024-01-21T04:33:50.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.391+0000 7fa1418b4640 1 -- start start 2024-01-21T04:33:50.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.391+0000 7fa1418b4640 10 monclient: init 2024-01-21T04:33:50.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-01-21T04:33:50.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa13c0b0f60 0x7fa13c0b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 10 monclient(hunting): picked mon.noname-b con 0x7fa13c0b0f60 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa13c0b3890 0x7fa13c0b2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 10 monclient(hunting): picked mon.noname-a con 0x7fa13c0b3890 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:50.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa13c1469b0 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:50.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa13c1473b0 con 0x7fa13c1469b0 2024-01-21T04:33:50.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa13c0b1340 con 0x7fa13c0b3890 2024-01-21T04:33:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c06ea10 con 0x7fa13c0b0f60 2024-01-21T04:33:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.392+0000 7fa1418b4640 10 monclient(hunting): authenticate will time out at 1388.734023s 2024-01-21T04:33:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.393+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5994007220 con 0x7f598c0047c0 2024-01-21T04:33:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.393+0000 7f59a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.393+0000 7f59a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.393+0000 7f59a2ffd640 10 monclient: dump: 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.393+0000 7f59aa9c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.393+0000 7f59a2ffd640 1 -- 172.21.15.69:0/1197830105 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599402e050 con 0x7f598c0047c0 2024-01-21T04:33:50.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.393+0000 7f59a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.393+0000 7f59aa9c0640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.393+0000 7f59aa9c0640 4 set_mon_vals no callback set 2024-01-21T04:33:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.394+0000 7f59aa9c0640 10 monclient: shutdown 2024-01-21T04:33:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.394+0000 7f59aa9c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.394+0000 7f59aa9c0640 1 -- 172.21.15.69:0/1197830105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f598c0047c0 msgr2=0x7f598c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.394+0000 7f59aa9c0640 1 --2- 172.21.15.69:0/1197830105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f598c0047c0 0x7f598c004c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f59a40b54d0 tx=0x7f598c0057f0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.394+0000 7f59aa9c0640 1 -- 172.21.15.69:0/1197830105 shutdown_connections 2024-01-21T04:33:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.394+0000 7f59aa9c0640 1 --2- 172.21.15.69:0/1197830105 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f598c0047c0 0x7f598c004c40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.394+0000 7f59aa9c0640 1 -- 172.21.15.69:0/1197830105 >> 172.21.15.69:0/1197830105 conn(0x7f59a4066fb0 msgr2=0x7f59a40b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.395+0000 7fa13b7fe640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa13c1469b0 0x7fa13c0b2850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58928/0 (socket says 172.21.15.69:58928) 2024-01-21T04:33:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.395+0000 7fa13b7fe640 1 -- 172.21.15.69:0/1931671255 learned_addr learned my addr 172.21.15.69:0/1931671255 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.395+0000 7fa133fff640 1 --2- 172.21.15.69:0/1931671255 >> v2:172.21.15.69:3300/0 conn(0x7fa13c0b3890 0x7fa13c0b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.395+0000 7fa133fff640 10 monclient(hunting): get_auth_request con 0x7fa13c0b3890 auth_method 0 2024-01-21T04:33:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.395+0000 7fa133fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.395+0000 7fa133fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.395+0000 7fa133fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.395+0000 7f59aa9c0640 1 -- 172.21.15.69:0/1197830105 shutdown_connections 2024-01-21T04:33:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.395+0000 7f59aa9c0640 1 -- 172.21.15.69:0/1197830105 wait complete. 2024-01-21T04:33:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.396+0000 7f59aa9c0640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.396+0000 7f59aa9c0640 10 monclient: monmap: 2024-01-21T04:33:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:50.401 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.396+0000 7fe3e97fa640 10 client.4289.objecter ms_handle_connect 0x7fe3d006c890 2024-01-21T04:33:50.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.396+0000 7fe3ebfff640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe3d006c890 0x7fe3d006ed50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec071d30 tx=0x7fe3d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.396+0000 7f59aa9c0640 1 librados: starting msgr at 2024-01-21T04:33:50.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.396+0000 7f59aa9c0640 1 librados: starting objecter 2024-01-21T04:33:50.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.396+0000 7f59aa9c0640 1 Processor -- start 2024-01-21T04:33:50.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7fa13affd640 1 --2- 172.21.15.69:0/1931671255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa13c0b0f60 0x7fa13c0b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 1 -- start start 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 1 librados: setting wanted keys 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 1 librados: calling monclient init 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 10 monclient: init 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59a40ac4e0 0x7f59a40acda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 10 monclient(hunting): picked mon.noname-b con 0x7f59a40ac4e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a40ac8c0 0x7f59a40ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 10 monclient(hunting): picked mon.noname-c con 0x7f59a40ac8c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a412fe70 0x7f59a41302e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 10 monclient(hunting): picked mon.noname-a con 0x7f59a412fe70 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a40ad820 con 0x7f59a412fe70 2024-01-21T04:33:50.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a40ae6d0 con 0x7f59a40ac4e0 2024-01-21T04:33:50.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a40b06b0 con 0x7f59a40ac8c0 2024-01-21T04:33:50.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.397+0000 7f59aa9c0640 10 monclient(hunting): authenticate will time out at 1388.739113s 2024-01-21T04:33:50.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 1 -- 172.21.15.69:0/2414602065 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec802ed50 con 0x7fced402b360 2024-01-21T04:33:50.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 10 monclient: dump: 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 1 -- 172.21.15.69:0/2414602065 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec8036440 con 0x7fced402b360 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 1 -- 172.21.15.69:0/2414602065 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec80367d0 con 0x7fced402b360 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcedb7fe640 10 monclient: dump: 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.409 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.410 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.410 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.410 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.410 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7f59a3fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59a40ac4e0 0x7f59a40acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcee1ea3640 5 monclient: authenticate success, global_id 4345 2024-01-21T04:33:50.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcee1ea3640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:50.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.399+0000 7fcee1ea3640 4 set_mon_vals no callback set 2024-01-21T04:33:50.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7f59a3fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59a40ac4e0 0x7f59a40acda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59802/0 (socket says 172.21.15.69:59802) 2024-01-21T04:33:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7f59a3fff640 1 -- 172.21.15.69:0/4106471582 learned_addr learned my addr 172.21.15.69:0/4106471582 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7f59a3fff640 10 monclient(hunting): get_auth_request con 0x7f59a40ac4e0 auth_method 0 2024-01-21T04:33:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7f59a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7f59a3fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7f59a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fcee1ea3640 10 monclient: shutdown 2024-01-21T04:33:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fcee1ea3640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fcee1ea3640 1 -- 172.21.15.69:0/2414602065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fced402b360 msgr2=0x7fced40c1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fcee1ea3640 1 --2- 172.21.15.69:0/2414602065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fced402b360 0x7fced40c1b50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcec800bb10 tx=0x7fcec802dfa0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fa133fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fa133fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fa133fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7f59a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7f59a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fcee1ea3640 1 -- 172.21.15.69:0/2414602065 shutdown_connections 2024-01-21T04:33:50.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fcee1ea3640 1 --2- 172.21.15.69:0/2414602065 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fced402b740 0x7fced40c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fcee1ea3640 1 --2- 172.21.15.69:0/2414602065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fced402b360 0x7fced40c1b50 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcec800bb10 tx=0x7fcec802dfa0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fcee1ea3640 1 -- 172.21.15.69:0/2414602065 >> 172.21.15.69:0/2414602065 conn(0x7fced403a920 msgr2=0x7fced403ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7f59a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fcee1ea3640 1 -- 172.21.15.69:0/2414602065 shutdown_connections 2024-01-21T04:33:50.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.400+0000 7fcee1ea3640 1 -- 172.21.15.69:0/2414602065 wait complete. 2024-01-21T04:33:50.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.401+0000 7fa13a7fc640 1 -- 172.21.15.69:0/1931671255 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887384519 0 0) 0x7fa13c1473b0 con 0x7fa13c1469b0 2024-01-21T04:33:50.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.401+0000 7fa13a7fc640 10 monclient(hunting): _negotiate 2024-01-21T04:33:50.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.401+0000 7fa13a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.401+0000 7fa13a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.401+0000 7fa13a7fc640 10 monclient(hunting): my global_id is 4317 2024-01-21T04:33:50.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.401+0000 7fa13a7fc640 1 -- 172.21.15.69:0/1931671255 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa128003520 con 0x7fa13c1469b0 2024-01-21T04:33:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.401+0000 7fcee1ea3640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.401+0000 7fcee1ea3640 10 monclient: monmap: 2024-01-21T04:33:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:50.418 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.402+0000 7fcee1ea3640 1 librados: starting msgr at 2024-01-21T04:33:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.402+0000 7fcee1ea3640 1 librados: starting objecter 2024-01-21T04:33:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.402+0000 7fcee1ea3640 1 Processor -- start 2024-01-21T04:33:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.402+0000 7fcee1ea3640 1 -- start start 2024-01-21T04:33:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.402+0000 7fcee1ea3640 1 librados: setting wanted keys 2024-01-21T04:33:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.402+0000 7fcee1ea3640 1 librados: calling monclient init 2024-01-21T04:33:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.402+0000 7fcee1ea3640 10 monclient: init 2024-01-21T04:33:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T04:33:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fced402b740 0x7fced40c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7f59a3fff640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2024-01-21T04:33:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7f59a3fff640 1 -- 172.21.15.69:0/4106471582 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a40ac8c0 msgr2=0x7f59a40ad2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7f59a3fff640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a40ac8c0 0x7f59a40ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7f59a3fff640 1 -- 172.21.15.69:0/4106471582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a412fe70 msgr2=0x7f59a41302e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7f59a3fff640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a412fe70 0x7f59a41302e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.404+0000 7f59a3fff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.404+0000 7f59a3fff640 1 monclient: found mon.noname-b 2024-01-21T04:33:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.404+0000 7f59a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.404+0000 7f59a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.404+0000 7f59a3fff640 1 -- 172.21.15.69:0/4106471582 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a40a9a90 con 0x7f59a40ac4e0 2024-01-21T04:33:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 10 monclient(hunting): picked mon.noname-c con 0x7fced402b740 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fced40c50e0 0x7fced402dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 10 monclient(hunting): picked mon.noname-b con 0x7fced40c50e0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fced415e930 0x7fced415edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 10 monclient(hunting): picked mon.noname-a con 0x7fced415e930 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced4039550 con 0x7fced415e930 2024-01-21T04:33:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced403a100 con 0x7fced40c50e0 2024-01-21T04:33:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced402f8c0 con 0x7fced402b740 2024-01-21T04:33:50.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.403+0000 7fcee1ea3640 10 monclient(hunting): authenticate will time out at 1388.745134s 2024-01-21T04:33:50.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.404+0000 7fa133fff640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2024-01-21T04:33:50.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.404+0000 7fa133fff640 1 -- 172.21.15.69:0/1931671255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa13c0b0f60 msgr2=0x7fa13c0b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.404+0000 7fa133fff640 1 --2- 172.21.15.69:0/1931671255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa13c0b0f60 0x7fa13c0b3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.404+0000 7fa133fff640 1 -- 172.21.15.69:0/1931671255 >> v1:172.21.15.69:6789/0 conn(0x7fa13c1469b0 legacy=0x7fa13c0b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.404+0000 7fa133fff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.404+0000 7fa133fff640 1 monclient: found mon.noname-a 2024-01-21T04:33:50.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.405+0000 7fa133fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.405+0000 7fa133fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.405+0000 7fa133fff640 1 -- 172.21.15.69:0/1931671255 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c06ecf0 con 0x7fa13c0b3890 2024-01-21T04:33:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.406+0000 7f5be2a90640 10 monclient: get_monmap_and_config 2024-01-21T04:33:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.406+0000 7f5be2a90640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.407+0000 7f5be2a90640 10 monclient: monmap: 2024-01-21T04:33:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:50.408000+0000 2024-01-21T04:33:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:50.408000+0000 2024-01-21T04:33:50.424 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.424 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.424 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:50.424 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:50.424 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:50.424 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:50.424 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:50.424 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.407+0000 7f59a0ff9640 10 client.?.objecter ms_handle_connect 0x7f59a40ac4e0 2024-01-21T04:33:50.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.407+0000 7f59a0ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.407+0000 7f59a3fff640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59a40ac4e0 0x7f59a40acda0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f599800b720 tx=0x7f599800bbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.408+0000 7f5be2a90640 1 Processor -- start 2024-01-21T04:33:50.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.408+0000 7f5be2a90640 1 -- start start 2024-01-21T04:33:50.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.408+0000 7f5be2a90640 10 monclient: init 2024-01-21T04:33:50.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.408+0000 7fcedbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fced40c50e0 0x7fced402dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-01-21T04:33:50.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bdc0a8b20 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:50.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bdc0a9d10 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:50.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f5bdc0b2310 0x7f5bdc13f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 10 monclient(hunting): picked mon.noname-c con 0x7f5bdc0b2310 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:50.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc1456f0 con 0x7f5bdc0a9d10 2024-01-21T04:33:50.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc0a98c0 con 0x7f5bdc0a8b20 2024-01-21T04:33:50.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7fcedbfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fced40c50e0 0x7fced402dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59804/0 (socket says 172.21.15.69:59804) 2024-01-21T04:33:50.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7fcedbfff640 1 -- 172.21.15.69:0/3496662790 learned_addr learned my addr 172.21.15.69:0/3496662790 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7fcedbfff640 10 monclient(hunting): get_auth_request con 0x7fced40c50e0 auth_method 0 2024-01-21T04:33:50.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7fcedbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bdc146290 con 0x7f5bdc0b2310 2024-01-21T04:33:50.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7f5be2a90640 10 monclient(hunting): authenticate will time out at 1388.750918s 2024-01-21T04:33:50.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7fcedbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7fcedbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.409+0000 7fa133fff640 1 --2- 172.21.15.69:0/1931671255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa13c0b3890 0x7fa13c0b2310 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa1240097a0 tx=0x7fa12402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa9bebc31ae1297 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7fcedbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7fcedbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7fcedbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59a0ff9640 1 -- 172.21.15.69:0/4106471582 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998010830 con 0x7f59a40ac4e0 2024-01-21T04:33:50.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59a0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59a0ff9640 10 monclient: dump: 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59a0ff9640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59a0ff9640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59aa9c0640 5 monclient: authenticate success, global_id 4348 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59aa9c0640 10 monclient: _renew_subs 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59aa9c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59aa9c0640 1 -- 172.21.15.69:0/4106471582 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a40ae940 con 0x7f59a40ac4e0 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59aa9c0640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59aa9c0640 10 monclient: _renew_subs 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59aa9c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59aa9c0640 1 -- 172.21.15.69:0/4106471582 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a4130a50 con 0x7f59a40ac4e0 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59aa9c0640 1 librados: init done 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59a0ff9640 1 -- 172.21.15.69:0/4106471582 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5998010990 con 0x7f59a40ac4e0 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.413+0000 7f59a27fc640 4 set_mon_vals no callback set 2024-01-21T04:33:50.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.414+0000 7f59a0ff9640 1 -- 172.21.15.69:0/4106471582 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998010830 con 0x7f59a40ac4e0 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.414+0000 7f59a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.414+0000 7f59a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.414+0000 7f59a0ff9640 10 monclient: dump: 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.433 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.415+0000 7f59aa9c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.415+0000 7f59aa9c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.415+0000 7f59aa9c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.415+0000 7f59aa9c0640 1 -- 172.21.15.69:0/4106471582 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5984005080 con 0x7f59a40ac4e0 2024-01-21T04:33:50.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7f5bdbfff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5bdc0a9d10 0x7f5bdc13f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58940/0 (socket says 172.21.15.69:58940) 2024-01-21T04:33:50.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcee3105640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fced415e930 0x7fced415edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7f5bdbfff640 1 -- 172.21.15.69:0/2667583414 learned_addr learned my addr 172.21.15.69:0/2667583414 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:50.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcee3105640 10 monclient(hunting): get_auth_request con 0x7fced415e930 auth_method 0 2024-01-21T04:33:50.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcee3105640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcee3105640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcee3105640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcedbfff640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2024-01-21T04:33:50.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcedbfff640 1 -- 172.21.15.69:0/3496662790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fced402b740 msgr2=0x7fced40c6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcedbfff640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fced402b740 0x7fced40c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcedbfff640 1 -- 172.21.15.69:0/3496662790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fced415e930 msgr2=0x7fced415edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcedbfff640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fced415e930 0x7fced415edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcedbfff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcedbfff640 1 monclient: found mon.noname-b 2024-01-21T04:33:50.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcedbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcedbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.416+0000 7fcedbfff640 1 -- 172.21.15.69:0/3496662790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec800b770 con 0x7fced40c50e0 2024-01-21T04:33:50.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 1 -- 172.21.15.69:0/1931671255 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa124031780 con 0x7fa13c0b3890 2024-01-21T04:33:50.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:50.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 10 monclient: dump: 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 1 -- 172.21.15.69:0/1931671255 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1240318e0 con 0x7fa13c0b3890 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 1 -- 172.21.15.69:0/1931671255 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa124031c70 con 0x7fa13c0b3890 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa13a7fc640 10 monclient: dump: 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa1418b4640 5 monclient: authenticate success, global_id 4314 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa1418b4640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.418+0000 7fa1418b4640 4 set_mon_vals no callback set 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.419+0000 7fa1418b4640 10 monclient: shutdown 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.419+0000 7fa1418b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.419+0000 7fa1418b4640 1 -- 172.21.15.69:0/1931671255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa13c0b3890 msgr2=0x7fa13c0b2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.419+0000 7fa1418b4640 1 --2- 172.21.15.69:0/1931671255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa13c0b3890 0x7fa13c0b2310 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa1240097a0 tx=0x7fa12402b820 comp rx=0 tx=0).stop 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.419+0000 7fa1418b4640 1 -- 172.21.15.69:0/1931671255 shutdown_connections 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.419+0000 7fa1418b4640 1 --2- 172.21.15.69:0/1931671255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa13c0b3890 0x7fa13c0b2310 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.419+0000 7fa1418b4640 1 --2- 172.21.15.69:0/1931671255 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa13c0b0f60 0x7fa13c0b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.419+0000 7fa1418b4640 1 -- 172.21.15.69:0/1931671255 >> 172.21.15.69:0/1931671255 conn(0x7fa13c066fb0 msgr2=0x7fa13c0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.419+0000 7fa1418b4640 1 -- 172.21.15.69:0/1931671255 shutdown_connections 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.419+0000 7fa1418b4640 1 -- 172.21.15.69:0/1931671255 wait complete. 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.420+0000 7fa1418b4640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.420+0000 7fa1418b4640 10 monclient: monmap: 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:50.443 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:50.444 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 librados: starting msgr at 2024-01-21T04:33:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 librados: starting objecter 2024-01-21T04:33:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 Processor -- start 2024-01-21T04:33:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 -- start start 2024-01-21T04:33:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 librados: setting wanted keys 2024-01-21T04:33:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 librados: calling monclient init 2024-01-21T04:33:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 10 monclient: init 2024-01-21T04:33:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:33:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa13c0b0f60 0x7fa13c14a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 10 monclient(hunting): picked mon.noname-a con 0x7fa13c0b0f60 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa13c0b3890 0x7fa13c140d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 10 monclient(hunting): picked mon.noname-b con 0x7fa13c0b3890 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa13c1469b0 0x7fa13c141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 10 monclient(hunting): picked mon.noname-c con 0x7fa13c1469b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c142430 con 0x7fa13c0b0f60 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c1496a0 con 0x7fa13c0b3890 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13c149910 con 0x7fa13c1469b0 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.421+0000 7fa1418b4640 10 monclient(hunting): authenticate will time out at 1388.763141s 2024-01-21T04:33:50.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.423+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410545358 0 0) 0x7f5bdc1456f0 con 0x7f5bdc0a9d10 2024-01-21T04:33:50.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.423+0000 7f5bdb7fe640 10 monclient(hunting): _negotiate 2024-01-21T04:33:50.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.423+0000 7f5bdb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.423+0000 7f5bdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.423+0000 7f5bdb7fe640 10 monclient(hunting): my global_id is 4323 2024-01-21T04:33:50.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.423+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc4003520 con 0x7f5bdc0a9d10 2024-01-21T04:33:50.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.425+0000 7fa13affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa13c0b0f60 0x7fa13c14a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7fa13affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa13c0b0f60 0x7fa13c14a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53378/0 (socket says 172.21.15.69:53378) 2024-01-21T04:33:50.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7fa13affd640 1 -- 172.21.15.69:0/1250008644 learned_addr learned my addr 172.21.15.69:0/1250008644 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7fa13affd640 10 monclient(hunting): get_auth_request con 0x7fa13c0b0f60 auth_method 0 2024-01-21T04:33:50.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7fa13affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7fa13affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7fa13affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4008976392 0 0) 0x7f5bc4003520 con 0x7f5bdc0a9d10 2024-01-21T04:33:50.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 >> v2:172.21.15.144:3300/0 conn(0x7f5bdc0b2310 msgr2=0x7f5bdc13f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 --2- 172.21.15.69:0/2667583414 >> v2:172.21.15.144:3300/0 conn(0x7f5bdc0b2310 0x7f5bdc13f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 >> v1:172.21.15.144:6789/0 conn(0x7f5bdc0a8b20 legacy=0x7f5bdc145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 monclient: found mon.noname-a-legacy 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.69:6789/0 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc1463d0 con 0x7f5bdc0a9d10 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 125097436 0 0) 0x7f5bd0001670 con 0x7f5bdc0a9d10 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient: dump: 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] but i'm connected to v1:172.21.15.69:6789/0, reconnecting 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 >> v1:172.21.15.69:6789/0 conn(0x7f5bdc0a9d10 legacy=0x7f5bdc13f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 --2- 172.21.15.69:0/2667583414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bc40039c0 0x7f5bc4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient(hunting): picked mon.c con 0x7f5bc40039c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 --2- 172.21.15.69:0/2667583414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc40047c0 0x7f5bc4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient(hunting): picked mon.b con 0x7f5bc40047c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 --2- 172.21.15.69:0/2667583414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc40052a0 0x7f5bc400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient(hunting): picked mon.a con 0x7f5bc40052a0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc146290 con 0x7f5bc40052a0 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc400c1f0 con 0x7f5bc40047c0 2024-01-21T04:33:50.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc400c3b0 con 0x7f5bc40039c0 2024-01-21T04:33:50.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.428+0000 7f5bdb7fe640 10 monclient(hunting): _finish_auth 0 2024-01-21T04:33:50.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.429+0000 7fa13affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.429+0000 7fa13affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.429+0000 7fa13affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.432+0000 7fa13affd640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.432+0000 7fa13affd640 1 -- 172.21.15.69:0/1250008644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa13c1469b0 msgr2=0x7fa13c141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.432+0000 7fa13affd640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa13c1469b0 0x7fa13c141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.432+0000 7fa13affd640 1 -- 172.21.15.69:0/1250008644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa13c0b3890 msgr2=0x7fa13c140d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.432+0000 7fa13affd640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa13c0b3890 0x7fa13c140d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.432+0000 7fa13affd640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.432+0000 7fa13affd640 1 monclient: found mon.noname-a 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.432+0000 7fa13affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.432+0000 7fa13affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.432+0000 7fa13affd640 1 -- 172.21.15.69:0/1250008644 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa124009400 con 0x7fa13c0b0f60 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.437+0000 7fa13affd640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa13c0b0f60 0x7fa13c14a0a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa12c00c820 tx=0x7fa12c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.437+0000 7fa138ff9640 10 client.?.objecter ms_handle_connect 0x7fa13c0b0f60 2024-01-21T04:33:50.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.437+0000 7fa138ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.439+0000 7f5be102d640 1 --2- 172.21.15.69:0/2667583414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc40052a0 0x7f5bc400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.440+0000 7f5be102d640 10 monclient(hunting): get_auth_request con 0x7f5bc40052a0 auth_method 0 2024-01-21T04:33:50.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.440+0000 7f5be102d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.440+0000 7f5be102d640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.440+0000 7f5be102d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T04:33:50.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7f5be102d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7f5be102d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7f5be102d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T04:33:50.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 1 -- 172.21.15.69:0/1250008644 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c004220 con 0x7fa13c0b0f60 2024-01-21T04:33:50.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 10 monclient: dump: 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 1 -- 172.21.15.69:0/1250008644 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12c0043a0 con 0x7fa13c0b0f60 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 1 -- 172.21.15.69:0/1250008644 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c007a70 con 0x7fa13c0b0f60 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa138ff9640 10 monclient: dump: 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.457 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa139ffb640 4 set_mon_vals no callback set 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa1418b4640 5 monclient: authenticate success, global_id 4326 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa1418b4640 10 monclient: _renew_subs 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa1418b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.441+0000 7fa1418b4640 1 -- 172.21.15.69:0/1250008644 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13c1509a0 con 0x7fa13c0b0f60 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.442+0000 7fa1418b4640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.442+0000 7fa1418b4640 10 monclient: _renew_subs 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.442+0000 7fa1418b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.442+0000 7fa1418b4640 1 -- 172.21.15.69:0/1250008644 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13c141790 con 0x7fa13c0b0f60 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.442+0000 7fa1418b4640 1 librados: init done 2024-01-21T04:33:50.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.443+0000 7fa1418b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.443+0000 7fa1418b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.443+0000 7fa1418b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.443+0000 7fa1418b4640 1 -- 172.21.15.69:0/1250008644 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa13c04e970 con 0x7fa13c0b0f60 2024-01-21T04:33:50.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 10 monclient(hunting): handle_auth_done global_id 4323 payload 293 2024-01-21T04:33:50.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 1 -- 172.21.15.69:0/2667583414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bc40039c0 msgr2=0x7f5bc4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 1 --2- 172.21.15.69:0/2667583414 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bc40039c0 0x7f5bc4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 1 -- 172.21.15.69:0/2667583414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc40047c0 msgr2=0x7f5bc4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 1 --2- 172.21.15.69:0/2667583414 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc40047c0 0x7f5bc4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 1 monclient: found mon.a 2024-01-21T04:33:50.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 1 -- 172.21.15.69:0/2667583414 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc4003520 con 0x7f5bc40052a0 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be2a90640 5 monclient: authenticate success, global_id 4323 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be2a90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.444+0000 7f5be102d640 1 -- 172.21.15.69:0/2667583414 reap_dead start 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.447+0000 7f5be102d640 1 --2- 172.21.15.69:0/2667583414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc40052a0 0x7f5bc400b980 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009e10 tx=0x7f5bc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf211547cdb1585 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.455+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd402cd50 con 0x7f5bc40052a0 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.455+0000 7f5bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.455+0000 7f5bdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.455+0000 7f5bdb7fe640 10 monclient: dump: 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.462 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.463 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.463 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.463 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.463 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.463 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.463 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.463 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.455+0000 7f5be2a90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:50.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.468+0000 7fa138ff9640 1 -- 172.21.15.69:0/1250008644 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa12c007bd0 con 0x7fa13c0b0f60 2024-01-21T04:33:50.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.469+0000 7fa138ff9640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa11c06c8e0 0x7fa11c06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.469+0000 7fa133fff640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa11c06c8e0 0x7fa11c06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.469+0000 7fa133fff640 10 monclient: get_auth_request con 0x7fa11c06c8e0 auth_method 0 2024-01-21T04:33:50.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.470+0000 7fa133fff640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa11c06c8e0 0x7fa11c06eda0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa124036000 tx=0x7fa124030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.470+0000 7fa138ff9640 10 client.4326.objecter ms_handle_connect 0x7fa11c06c8e0 2024-01-21T04:33:50.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.470+0000 7fe3e97fa640 1 -- 172.21.15.69:0/3704714150 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe3e002d040 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.470+0000 7fe3e97fa640 10 client.4289.objecter ms_dispatch 0x7fe3ec000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:50.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.470+0000 7fe3e97fa640 3 client.4289.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:50.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.470+0000 7fe3e97fa640 3 client.4289.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:50.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.471+0000 7fe3e97fa640 20 client.4289.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.472+0000 7fe4ce4de640 10 monclient: get_monmap_and_config 2024-01-21T04:33:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.472+0000 7fe4ce4de640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.472+0000 7fe4ce4de640 10 monclient: monmap: 2024-01-21T04:33:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:50.473239+0000 2024-01-21T04:33:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:50.473239+0000 2024-01-21T04:33:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:50.473 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:50.474 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:50.474 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:50.474 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:50.474 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 1 Processor -- start 2024-01-21T04:33:50.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 1 -- start start 2024-01-21T04:33:50.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 10 monclient: init 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c80a8b20 0x7fe4c8145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c80a8b20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe4c80a9d10 0x7fe4c813f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c80a9d10 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4c80b2310 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c81456f0 con 0x7fe4c80b2310 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c80a98c0 con 0x7fe4c80a9d10 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c81460c0 con 0x7fe4c80a8b20 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.473+0000 7fe4ce4de640 10 monclient(hunting): authenticate will time out at 1388.815199s 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.484+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd4003990 con 0x7f5bc40052a0 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.484+0000 7f5bdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.484+0000 7f5bdb7fe640 1 -- 172.21.15.69:0/2667583414 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4003d20 con 0x7f5bc40052a0 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.484+0000 7f5bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.484+0000 7f5bdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.484+0000 7f5bdb7fe640 10 monclient: dump: 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.486 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.487 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.487 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.487 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.487 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.484+0000 7f5be2a90640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:50.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.484+0000 7f5be2a90640 4 set_mon_vals no callback set 2024-01-21T04:33:50.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.484+0000 7f8a7d93e640 10 monclient: get_monmap_and_config 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.484+0000 7f8a7d93e640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.484+0000 7f8a7d93e640 10 monclient: monmap: 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:50.485681+0000 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:50.485681+0000 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:50.488 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7fa138ff9640 1 -- 172.21.15.69:0/1250008644 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa12c058ac0 con 0x7fa13c0b0f60 2024-01-21T04:33:50.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7fa138ff9640 10 client.4326.objecter ms_dispatch 0x7fa13c000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:50.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7fa138ff9640 3 client.4326.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:50.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7fa138ff9640 3 client.4326.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:50.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7fa138ff9640 20 client.4326.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f5be2a90640 10 monclient: shutdown 2024-01-21T04:33:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f5be2a90640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f5be2a90640 1 -- 172.21.15.69:0/2667583414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc40052a0 msgr2=0x7f5bc400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f5be2a90640 1 --2- 172.21.15.69:0/2667583414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc40052a0 0x7f5bc400b980 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009e10 tx=0x7f5bc40057f0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f5be2a90640 1 -- 172.21.15.69:0/2667583414 shutdown_connections 2024-01-21T04:33:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f5be2a90640 1 --2- 172.21.15.69:0/2667583414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bc40052a0 0x7f5bc400b980 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f5be2a90640 1 -- 172.21.15.69:0/2667583414 >> 172.21.15.69:0/2667583414 conn(0x7f5bdc066fb0 msgr2=0x7f5bdc0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f5be2a90640 1 -- 172.21.15.69:0/2667583414 shutdown_connections 2024-01-21T04:33:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f5be2a90640 1 -- 172.21.15.69:0/2667583414 wait complete. 2024-01-21T04:33:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f8a7d93e640 1 Processor -- start 2024-01-21T04:33:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f8a7d93e640 1 -- start start 2024-01-21T04:33:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.485+0000 7f8a7d93e640 10 monclient: init 2024-01-21T04:33:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-01-21T04:33:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8a780a8b20 0x7f8a78145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 10 monclient(hunting): picked mon.noname-c con 0x7f8a780a8b20 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a780a9d10 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a780b2310 0x7f8a7813f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 10 monclient(hunting): picked mon.noname-a con 0x7f8a780b2310 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a781456f0 con 0x7f8a780a9d10 2024-01-21T04:33:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a780a98c0 con 0x7f8a780b2310 2024-01-21T04:33:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a781460c0 con 0x7f8a780a8b20 2024-01-21T04:33:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f8a7d93e640 10 monclient(hunting): authenticate will time out at 1388.828002s 2024-01-21T04:33:50.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f5be2a90640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.486+0000 7f5be2a90640 10 monclient: monmap: 2024-01-21T04:33:50.496 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.496 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:50.496 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:50.496 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:50.496 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.496 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.496 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:50.496 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:50.496 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:50.496 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.487+0000 7f5be2a90640 1 librados: starting msgr at 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.487+0000 7f5be2a90640 1 librados: starting objecter 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.487+0000 7f5be2a90640 1 Processor -- start 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.487+0000 7f5be2a90640 1 -- start start 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.487+0000 7f5be2a90640 1 librados: setting wanted keys 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.487+0000 7f5be2a90640 1 librados: calling monclient init 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.487+0000 7f5be2a90640 10 monclient: init 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.488+0000 7f5be2a90640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.488+0000 7f5be2a90640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.488+0000 7f5be2a90640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc40052a0 0x7f5bdc0abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 10 monclient(hunting): picked mon.noname-b con 0x7f5bc40052a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bdc1d7470 0x7f5bdc1d78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 10 monclient(hunting): picked mon.noname-a con 0x7f5bdc1d7470 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bdc137d60 0x7f5bdc1381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 10 monclient(hunting): picked mon.noname-c con 0x7f5bdc137d60 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc0ab930 con 0x7f5bdc1d7470 2024-01-21T04:33:50.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc142c00 con 0x7f5bc40052a0 2024-01-21T04:33:50.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc0ad380 con 0x7f5bdc137d60 2024-01-21T04:33:50.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.489+0000 7f5be2a90640 10 monclient(hunting): authenticate will time out at 1388.830630s 2024-01-21T04:33:50.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff31effd640 1 -- 172.21.15.69:0/4075847385 <== osd.3 v2:172.21.15.69:6800/1086763384 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff32802b9f0 con 0x7ff310072c40 2024-01-21T04:33:50.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff31effd640 10 client.4277.objecter ms_dispatch 0x7ff330000b90 command_reply(tid 2: 0 ) v1 2024-01-21T04:33:50.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff31effd640 10 client.4277.objecter _finish_command 2 = system:0 2024-01-21T04:33:50.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff31effd640 20 client.4277.objecter put_session s=0x7ff310072570 osd=3 4 2024-01-21T04:33:50.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff31effd640 15 client.4277.objecter _session_command_op_remove 3 2 2024-01-21T04:33:50.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 10 librados: watch_flush enter 2024-01-21T04:33:50.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 10 librados: watch_flush exit 2024-01-21T04:33:50.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 10 client.4277.objecter close_session for osd.3 2024-01-21T04:33:50.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 1 -- 172.21.15.69:0/4075847385 >> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] conn(0x7ff310072c40 msgr2=0x7ff310075060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] conn(0x7ff310072c40 0x7ff310075060 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 20 client.4277.objecter put_session s=0x7ff310072570 osd=3 1 2024-01-21T04:33:50.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 20 client.4277.objecter shutdown clearing up homeless session... 2024-01-21T04:33:50.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 10 client.4277.objecter successfully canceled tick 2024-01-21T04:33:50.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 1 -- 172.21.15.69:0/4075847385 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7ff31006c860 msgr2=0x7ff31006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7ff31006c860 0x7ff31006ed20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff320001560 tx=0x7ff320007990 comp rx=0 tx=0).stop 2024-01-21T04:33:50.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 10 monclient: shutdown 2024-01-21T04:33:50.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 1 -- 172.21.15.69:0/4075847385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3280322a0 msgr2=0x7ff3280c08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 1 --2- 172.21.15.69:0/4075847385 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff3280322a0 0x7ff3280c08b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff324000c00 tx=0x7ff3240046c0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff32ed9d640 1 -- 172.21.15.69:0/4075847385 reap_dead start 2024-01-21T04:33:50.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 1 -- 172.21.15.69:0/4075847385 shutdown_connections 2024-01-21T04:33:50.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 1 -- 172.21.15.69:0/4075847385 >> 172.21.15.69:0/4075847385 conn(0x7ff328006af0 msgr2=0x7ff328037960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.534+0000 7ff334d0f640 1 -- 172.21.15.69:0/4075847385 shutdown_connections 2024-01-21T04:33:50.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.535+0000 7ff334d0f640 1 -- 172.21.15.69:0/4075847385 wait complete. 2024-01-21T04:33:50.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.535+0000 7ff334d0f640 1 librados: shutdown 2024-01-21T04:33:50.549 INFO:teuthology.orchestra.run.smithi069.stdout:90194313226 2024-01-21T04:33:50.550 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-21T04:33:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.680+0000 7fe3e97fa640 1 -- 172.21.15.69:0/3704714150 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe3e0077080 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.680+0000 7fe3e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.680+0000 7fe3e97fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:50.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.683+0000 7fe4c7fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c80a8b20 0x7fe4c8145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.683+0000 7fe4c7fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c80a8b20 0x7fe4c8145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46946/0 (socket says 172.21.15.69:46946) 2024-01-21T04:33:50.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.683+0000 7fe4c7fff640 1 -- 172.21.15.69:0/1075796318 learned_addr learned my addr 172.21.15.69:0/1075796318 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.683+0000 7fe4c7fff640 10 monclient(hunting): get_auth_request con 0x7fe4c80a8b20 auth_method 0 2024-01-21T04:33:50.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.683+0000 7fe4c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.683+0000 7fe4c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.683+0000 7fe4c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.685+0000 7f5be102d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bdc137d60 0x7f5bdc1381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.686+0000 7f5be102d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bdc137d60 0x7f5bdc1381e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46948/0 (socket says 172.21.15.69:46948) 2024-01-21T04:33:50.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.686+0000 7f5be102d640 1 -- 172.21.15.69:0/1612289380 learned_addr learned my addr 172.21.15.69:0/1612289380 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.686+0000 7f5be102d640 10 monclient(hunting): get_auth_request con 0x7f5bdc137d60 auth_method 0 2024-01-21T04:33:50.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.686+0000 7f5be102d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.686+0000 7f5be102d640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.686+0000 7f5be102d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.687+0000 7fe4c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.687+0000 7fe4c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.687+0000 7fe4c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.688+0000 7f5be102d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.688+0000 7f5be102d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.688+0000 7f5be102d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.689+0000 7f59a0ff9640 1 -- 172.21.15.69:0/4106471582 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5998010b00 con 0x7f59a40ac4e0 2024-01-21T04:33:50.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.689+0000 7f59a0ff9640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f599006c950 0x7f599006ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.689+0000 7f59a37fe640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f599006c950 0x7f599006ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.690+0000 7f59a37fe640 10 monclient: get_auth_request con 0x7f599006c950 auth_method 0 2024-01-21T04:33:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.690+0000 7f59a0ff9640 10 client.4348.objecter ms_handle_connect 0x7f599006c950 2024-01-21T04:33:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.690+0000 7f59a37fe640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f599006c950 0x7f599006ee10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5994033000 tx=0x7f599402f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.691+0000 7fe4c7fff640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-01-21T04:33:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.691+0000 7fe4c7fff640 1 -- 172.21.15.69:0/1075796318 >> v2:172.21.15.144:3300/0 conn(0x7fe4c80a9d10 msgr2=0x7fe4c813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.691+0000 7fe4c7fff640 1 --2- 172.21.15.69:0/1075796318 >> v2:172.21.15.144:3300/0 conn(0x7fe4c80a9d10 0x7fe4c813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.691+0000 7fe4c7fff640 1 -- 172.21.15.69:0/1075796318 >> v1:172.21.15.69:6789/0 conn(0x7fe4c80b2310 legacy=0x7fe4c813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.691+0000 7fe4c7fff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.691+0000 7fe4c7fff640 1 monclient: found mon.noname-b 2024-01-21T04:33:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.691+0000 7fe4c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.691+0000 7fe4c7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.691+0000 7fe4c7fff640 1 -- 172.21.15.69:0/1075796318 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c8146230 con 0x7fe4c80a8b20 2024-01-21T04:33:50.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.692+0000 7f59a0ff9640 1 -- 172.21.15.69:0/4106471582 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5998058e50 con 0x7f59a40ac4e0 2024-01-21T04:33:50.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.692+0000 7f59a0ff9640 10 client.4348.objecter ms_dispatch 0x7f59a4000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:50.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.692+0000 7f59a0ff9640 3 client.4348.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:50.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.692+0000 7f59a0ff9640 3 client.4348.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:50.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.692+0000 7f59a0ff9640 20 client.4348.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.692+0000 7fe12c73e640 10 monclient: get_monmap_and_config 2024-01-21T04:33:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.692+0000 7fe12c73e640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.692+0000 7fe12c73e640 10 monclient: monmap: 2024-01-21T04:33:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:50.693930+0000 2024-01-21T04:33:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:50.693930+0000 2024-01-21T04:33:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.695 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.695 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:50.695 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:50.695 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:50.695 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:50.695 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:50.695 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7fe12c73e640 1 Processor -- start 2024-01-21T04:33:50.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7fe12c73e640 1 -- start start 2024-01-21T04:33:50.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7fe12c73e640 10 monclient: init 2024-01-21T04:33:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7f5be102d640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-01-21T04:33:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7f5be102d640 1 -- 172.21.15.69:0/1612289380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc40052a0 msgr2=0x7f5bdc0abda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7f5be102d640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc40052a0 0x7f5bdc0abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7f5be102d640 1 -- 172.21.15.69:0/1612289380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bdc1d7470 msgr2=0x7f5bdc1d78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7f5be102d640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bdc1d7470 0x7f5bdc1d78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7f5be102d640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7f5be102d640 1 monclient: found mon.noname-c 2024-01-21T04:33:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7f5be102d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7f5be102d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.693+0000 7f5be102d640 1 -- 172.21.15.69:0/1612289380 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc0ad660 con 0x7f5bdc137d60 2024-01-21T04:33:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-01-21T04:33:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe1240a8b20 0x7fe124145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 10 monclient(hunting): picked mon.noname-c con 0x7fe1240a8b20 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe1240a9d10 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe1240b2310 0x7fe12413f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 10 monclient(hunting): picked mon.noname-a con 0x7fe1240b2310 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1241456f0 con 0x7fe1240a9d10 2024-01-21T04:33:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1240a98c0 con 0x7fe1240b2310 2024-01-21T04:33:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1241460c0 con 0x7fe1240a8b20 2024-01-21T04:33:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.694+0000 7fe12c73e640 10 monclient(hunting): authenticate will time out at 1389.035602s 2024-01-21T04:33:50.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.697+0000 7fe4c7fff640 1 --2- 172.21.15.69:0/1075796318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c80a8b20 0x7fe4c8145b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009870 tx=0x7fe4b802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6903655d1812370 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.698+0000 7f5be102d640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bdc137d60 0x7f5bdc1381e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009f30 tx=0x7f5bd4004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.698+0000 7f5bd97fa640 10 client.?.objecter ms_handle_connect 0x7f5bdc137d60 2024-01-21T04:33:50.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.698+0000 7f5bd97fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.702+0000 7f59a0ff9640 1 -- 172.21.15.69:0/4106471582 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5998054a90 con 0x7f59a40ac4e0 2024-01-21T04:33:50.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.702+0000 7f59a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.702+0000 7f59a0ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.704+0000 7fcedbfff640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fced40c50e0 0x7fced402dce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fced000b4a0 tx=0x7fced000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.704+0000 7fced97fa640 10 client.?.objecter ms_handle_connect 0x7fced40c50e0 2024-01-21T04:33:50.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.704+0000 7fced97fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 1 -- 172.21.15.69:0/1075796318 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b80317c0 con 0x7fe4c80a8b20 2024-01-21T04:33:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 10 monclient: dump: 2024-01-21T04:33:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 1 -- 172.21.15.69:0/1075796318 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b8031920 con 0x7fe4c80a8b20 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4ce4de640 5 monclient: authenticate success, global_id 4292 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4ce4de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4ce4de640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 1 -- 172.21.15.69:0/1075796318 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8031cb0 con 0x7fe4c80a8b20 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4ce4de640 4 set_mon_vals no callback set 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4c6ffd640 10 monclient: dump: 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.710 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4ce4de640 10 monclient: shutdown 2024-01-21T04:33:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4ce4de640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1075796318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c80a8b20 msgr2=0x7fe4c8145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.706+0000 7fe4ce4de640 1 --2- 172.21.15.69:0/1075796318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c80a8b20 0x7fe4c8145b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009870 tx=0x7fe4b802b860 comp rx=0 tx=0).stop 2024-01-21T04:33:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1075796318 shutdown_connections 2024-01-21T04:33:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7fe4ce4de640 1 --2- 172.21.15.69:0/1075796318 >> v2:172.21.15.144:3300/0 conn(0x7fe4c80a9d10 0x7fe4c813f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7fe4ce4de640 1 --2- 172.21.15.69:0/1075796318 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c80a8b20 0x7fe4c8145b80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1075796318 >> 172.21.15.69:0/1075796318 conn(0x7fe4c8066fb0 msgr2=0x7fe4c80b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1075796318 shutdown_connections 2024-01-21T04:33:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1075796318 wait complete. 2024-01-21T04:33:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 1 -- 172.21.15.69:0/1612289380 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4036810 con 0x7f5bdc137d60 2024-01-21T04:33:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 10 monclient: dump: 2024-01-21T04:33:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.713 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.713 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.713 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.713 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.713 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 1 -- 172.21.15.69:0/1612289380 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd4036970 con 0x7f5bdc137d60 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 1 -- 172.21.15.69:0/1612289380 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4036d60 con 0x7f5bdc137d60 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bd97fa640 10 monclient: dump: 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5bdaffd640 4 set_mon_vals no callback set 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5be2a90640 5 monclient: authenticate success, global_id 4295 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5be2a90640 10 monclient: _renew_subs 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5be2a90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5be2a90640 1 -- 172.21.15.69:0/1612289380 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc1d7e10 con 0x7f5bdc137d60 2024-01-21T04:33:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5be2a90640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5be2a90640 10 monclient: _renew_subs 2024-01-21T04:33:50.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5be2a90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5be2a90640 1 -- 172.21.15.69:0/1612289380 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc138720 con 0x7f5bdc137d60 2024-01-21T04:33:50.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.707+0000 7f5be2a90640 1 librados: init done 2024-01-21T04:33:50.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.708+0000 7fe4ce4de640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.708+0000 7fe4ce4de640 10 monclient: monmap: 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.708+0000 7fe4ce4de640 1 librados: starting msgr at 2024-01-21T04:33:50.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.708+0000 7fe4ce4de640 1 librados: starting objecter 2024-01-21T04:33:50.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.708+0000 7f5be2a90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.708+0000 7f5be2a90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.708+0000 7f5be2a90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.708+0000 7fe4ce4de640 1 Processor -- start 2024-01-21T04:33:50.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.708+0000 7f5be2a90640 1 -- 172.21.15.69:0/1612289380 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bdc04e970 con 0x7f5bdc137d60 2024-01-21T04:33:50.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 1 -- start start 2024-01-21T04:33:50.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 1 librados: setting wanted keys 2024-01-21T04:33:50.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 1 librados: calling monclient init 2024-01-21T04:33:50.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 10 monclient: init 2024-01-21T04:33:50.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:33:50.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4c80a8b20 0x7fe4c8142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c80a8b20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4c80a9d10 0x7fe4c8138040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c80a9d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c80b2310 0x7fe4c813a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c80b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c813b5b0 con 0x7fe4c80a8b20 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c8138580 con 0x7fe4c80a9d10 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c81395f0 con 0x7fe4c80b2310 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.709+0000 7fe4ce4de640 10 monclient(hunting): authenticate will time out at 1389.051165s 2024-01-21T04:33:50.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.712+0000 7f8a76ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8a780a8b20 0x7f8a78145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.713+0000 7f8a76ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8a780a8b20 0x7f8a78145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59844/0 (socket says 172.21.15.69:59844) 2024-01-21T04:33:50.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.713+0000 7f8a76ffd640 1 -- 172.21.15.69:0/3749889605 learned_addr learned my addr 172.21.15.69:0/3749889605 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.713+0000 7f8a76ffd640 10 monclient(hunting): get_auth_request con 0x7f8a780a8b20 auth_method 0 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.713+0000 7f8a76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.713+0000 7f8a76ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.713+0000 7f8a76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.714+0000 7f8a76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.714+0000 7f8a76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.714+0000 7f8a76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.716+0000 7f8a76ffd640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.716+0000 7f8a76ffd640 1 -- 172.21.15.69:0/3749889605 >> v2:172.21.15.69:3300/0 conn(0x7f8a780b2310 msgr2=0x7f8a7813f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.716+0000 7f8a76ffd640 1 --2- 172.21.15.69:0/3749889605 >> v2:172.21.15.69:3300/0 conn(0x7f8a780b2310 0x7f8a7813f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.716+0000 7f8a76ffd640 1 -- 172.21.15.69:0/3749889605 >> v1:172.21.15.144:6789/0 conn(0x7f8a780a9d10 legacy=0x7f8a7813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.716+0000 7f8a76ffd640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.716+0000 7f8a76ffd640 1 monclient: found mon.noname-c 2024-01-21T04:33:50.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.716+0000 7f8a76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.716+0000 7f8a76ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.716+0000 7f8a76ffd640 1 -- 172.21.15.69:0/3749889605 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78146230 con 0x7f8a780a8b20 2024-01-21T04:33:50.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.717+0000 7fe12a4da640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe1240a8b20 0x7fe124145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fe4c77fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4c80a9d10 0x7fe4c8138040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 1 -- 172.21.15.69:0/3496662790 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0010610 con 0x7fced40c50e0 2024-01-21T04:33:50.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 10 monclient: dump: 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 1 -- 172.21.15.69:0/3496662790 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced0010770 con 0x7fced40c50e0 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 1 -- 172.21.15.69:0/3496662790 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0010b20 con 0x7fced40c50e0 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fced97fa640 10 monclient: dump: 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.727 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.728 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fcee1ea3640 5 monclient: authenticate success, global_id 4351 2024-01-21T04:33:50.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fcedaffd640 4 set_mon_vals no callback set 2024-01-21T04:33:50.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.718+0000 7fcee1ea3640 10 monclient: _renew_subs 2024-01-21T04:33:50.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fcee1ea3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fcee1ea3640 1 -- 172.21.15.69:0/3496662790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced4039be0 con 0x7fced40c50e0 2024-01-21T04:33:50.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fcee1ea3640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fcee1ea3640 10 monclient: _renew_subs 2024-01-21T04:33:50.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fcee1ea3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fcee1ea3640 1 -- 172.21.15.69:0/3496662790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced4164b60 con 0x7fced40c50e0 2024-01-21T04:33:50.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe12a4da640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fe1240a8b20 0x7fe124145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59862/0 (socket says 172.21.15.69:59862) 2024-01-21T04:33:50.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe12a4da640 1 -- 172.21.15.69:0/2108966769 learned_addr learned my addr 172.21.15.69:0/2108966769 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe12a4da640 10 monclient(hunting): get_auth_request con 0x7fe1240a8b20 auth_method 0 2024-01-21T04:33:50.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe12a4da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe12a4da640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe12a4da640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fcee1ea3640 1 librados: init done 2024-01-21T04:33:50.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe4c77fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4c80a9d10 0x7fe4c8138040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59870/0 (socket says 172.21.15.69:59870) 2024-01-21T04:33:50.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe4c77fe640 1 -- 172.21.15.69:0/1933384203 learned_addr learned my addr 172.21.15.69:0/1933384203 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request con 0x7fe4c80a9d10 auth_method 0 2024-01-21T04:33:50.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe4c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe4c77fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.719+0000 7fe4c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.720+0000 7fcee1ea3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.720+0000 7fcee1ea3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.720+0000 7fcee1ea3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.720+0000 7fcee1ea3640 1 -- 172.21.15.69:0/3496662790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced4039dd0 con 0x7fced40c50e0 2024-01-21T04:33:50.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.722+0000 7f8a76ffd640 1 --2- 172.21.15.69:0/3749889605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a780a8b20 0x7f8a78145b80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8a780a8600 tx=0x7f8a6802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=169c430d7f7ba33f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.735+0000 7fe12acdb640 1 --2- 172.21.15.69:0/2108966769 >> v2:172.21.15.69:3300/0 conn(0x7fe1240b2310 0x7fe12413f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.736+0000 7fe12acdb640 10 monclient(hunting): get_auth_request con 0x7fe1240b2310 auth_method 0 2024-01-21T04:33:50.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.736+0000 7fe12acdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.736+0000 7fe12acdb640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.736+0000 7fe12acdb640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.742+0000 7fced97fa640 1 -- 172.21.15.69:0/3496662790 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fced0010c80 con 0x7fced40c50e0 2024-01-21T04:33:50.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.742+0000 7f5bd97fa640 1 -- 172.21.15.69:0/1612289380 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5bd4034440 con 0x7f5bdc137d60 2024-01-21T04:33:50.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.742+0000 7fced97fa640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcec006c8b0 0x7fcec006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.742+0000 7f5bd97fa640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5bc806c890 0x7f5bc806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.743+0000 7fcee0ea1640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcec006c8b0 0x7fcec006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.743+0000 7fcee0ea1640 10 monclient: get_auth_request con 0x7fcec006c8b0 auth_method 0 2024-01-21T04:33:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.743+0000 7f5be082c640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5bc806c890 0x7f5bc806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.743+0000 7f5be082c640 10 monclient: get_auth_request con 0x7f5bc806c890 auth_method 0 2024-01-21T04:33:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.743+0000 7f5be082c640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5bc806c890 0x7f5bc806ed50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0014e0 tx=0x7f5bcc007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.743+0000 7f5bd97fa640 10 client.4295.objecter ms_handle_connect 0x7f5bc806c890 2024-01-21T04:33:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.743+0000 7fcee0ea1640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcec006c8b0 0x7fcec006ed70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcec8038000 tx=0x7fcec8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.743+0000 7fced97fa640 10 client.4351.objecter ms_handle_connect 0x7fcec006c8b0 2024-01-21T04:33:50.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.747+0000 7f5bd97fa640 1 -- 172.21.15.69:0/1612289380 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5bd407b440 con 0x7f5bdc137d60 2024-01-21T04:33:50.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.747+0000 7f5bd97fa640 10 client.4295.objecter ms_dispatch 0x7f5bdc000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:50.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.747+0000 7f5bd97fa640 3 client.4295.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:50.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.747+0000 7f5bd97fa640 3 client.4295.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:50.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.747+0000 7f5bd97fa640 20 client.4295.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.748+0000 7fa138ff9640 1 -- 172.21.15.69:0/1250008644 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa12c054700 con 0x7fa13c0b0f60 2024-01-21T04:33:50.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.748+0000 7fa138ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.748+0000 7fa138ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:50.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.748+0000 7fe12acdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.748+0000 7fe12acdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.748+0000 7fe12acdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.749+0000 7fe4c7fff640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4c80a8b20 0x7fe4c8142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.751+0000 7fe12acdb640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2024-01-21T04:33:50.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.751+0000 7fe12acdb640 1 -- 172.21.15.69:0/2108966769 >> v2:172.21.15.144:3300/0 conn(0x7fe1240a8b20 msgr2=0x7fe124145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:50.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.751+0000 7fe12acdb640 1 --2- 172.21.15.69:0/2108966769 >> v2:172.21.15.144:3300/0 conn(0x7fe1240a8b20 0x7fe124145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.751+0000 7fe12acdb640 1 -- 172.21.15.69:0/2108966769 >> v1:172.21.15.69:6789/0 conn(0x7fe1240a9d10 legacy=0x7fe12413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.751+0000 7fe12acdb640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.751+0000 7fe12acdb640 1 monclient: found mon.noname-a 2024-01-21T04:33:50.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.751+0000 7fe12acdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.751+0000 7fe12acdb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.751+0000 7fe12acdb640 1 -- 172.21.15.69:0/2108966769 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe124146230 con 0x7fe1240b2310 2024-01-21T04:33:50.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.753+0000 7fe4c7fff640 10 monclient(hunting): get_auth_request con 0x7fe4c80a8b20 auth_method 0 2024-01-21T04:33:50.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.753+0000 7fe4c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.753+0000 7fe4c7fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.753+0000 7fe4c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.757+0000 7fe12acdb640 1 --2- 172.21.15.69:0/2108966769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1240b2310 0x7fe12413f790 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe1240a8600 tx=0x7fe11c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c80c5a162909bd server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.758+0000 7fe4c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.758+0000 7fe4c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.758+0000 7fe4c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.762+0000 7fe4c7fff640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2024-01-21T04:33:50.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.762+0000 7fe4c7fff640 1 -- 172.21.15.69:0/1933384203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c80b2310 msgr2=0x7fe4c813a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.762+0000 7fe4c7fff640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c80b2310 0x7fe4c813a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.762+0000 7fe4c7fff640 1 -- 172.21.15.69:0/1933384203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4c80a9d10 msgr2=0x7fe4c8138040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.762+0000 7fe4c7fff640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4c80a9d10 0x7fe4c8138040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.762+0000 7fe4c7fff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.762+0000 7fe4c7fff640 1 monclient: found mon.noname-a 2024-01-21T04:33:50.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.762+0000 7fe4c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.762+0000 7fe4c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.762+0000 7fe4c7fff640 1 -- 172.21.15.69:0/1933384203 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b80094d0 con 0x7fe4c80a8b20 2024-01-21T04:33:50.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.768+0000 7fe4c7fff640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4c80a8b20 0x7fe4c8142930 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8033000 tx=0x7fe4b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.768+0000 7fe4c4ff9640 10 client.?.objecter ms_handle_connect 0x7fe4c80a8b20 2024-01-21T04:33:50.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.768+0000 7fe4c4ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 1 -- 172.21.15.69:0/2108966769 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c02cc70 con 0x7fe1240b2310 2024-01-21T04:33:50.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:50.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 10 monclient: dump: 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 1 -- 172.21.15.69:0/2108966769 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11c02cdd0 con 0x7fe1240b2310 2024-01-21T04:33:50.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 1 -- 172.21.15.69:0/2108966769 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11c034660 con 0x7fe1240b2310 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe1294d8640 10 monclient: dump: 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe12c73e640 5 monclient: authenticate success, global_id 4329 2024-01-21T04:33:50.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe12c73e640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:50.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.771+0000 7fe12c73e640 4 set_mon_vals no callback set 2024-01-21T04:33:50.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.772+0000 7fe12c73e640 10 monclient: shutdown 2024-01-21T04:33:50.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.772+0000 7fe12c73e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.772+0000 7fe12c73e640 1 -- 172.21.15.69:0/2108966769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1240b2310 msgr2=0x7fe12413f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.772+0000 7fe12c73e640 1 --2- 172.21.15.69:0/2108966769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1240b2310 0x7fe12413f790 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe1240a8600 tx=0x7fe11c02bec0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.772+0000 7fe12c73e640 1 -- 172.21.15.69:0/2108966769 shutdown_connections 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.772+0000 7fe12c73e640 1 --2- 172.21.15.69:0/2108966769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1240b2310 0x7fe12413f790 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.772+0000 7fe12c73e640 1 --2- 172.21.15.69:0/2108966769 >> v2:172.21.15.144:3300/0 conn(0x7fe1240a8b20 0x7fe124145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.772+0000 7fe12c73e640 1 -- 172.21.15.69:0/2108966769 >> 172.21.15.69:0/2108966769 conn(0x7fe124066fb0 msgr2=0x7fe1240b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.772+0000 7fe12c73e640 1 -- 172.21.15.69:0/2108966769 shutdown_connections 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.772+0000 7fe12c73e640 1 -- 172.21.15.69:0/2108966769 wait complete. 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.773+0000 7fe12c73e640 10 monclient: build_initial_monmap 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.773+0000 7fe12c73e640 10 monclient: monmap: 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:50.777 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.774+0000 7fe12c73e640 1 librados: starting msgr at 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.774+0000 7fe12c73e640 1 librados: starting objecter 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.774+0000 7fe12c73e640 1 Processor -- start 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.774+0000 7fe12c73e640 1 -- start start 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.775+0000 7fe12c73e640 1 librados: setting wanted keys 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.775+0000 7fe12c73e640 1 librados: calling monclient init 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.775+0000 7fe12c73e640 10 monclient: init 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:50.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:33:50.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1240a8b20 0x7fe1241380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 10 monclient(hunting): picked mon.noname-a con 0x7fe1240a8b20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:50.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1240a9d10 0x7fe124133a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 10 monclient(hunting): picked mon.noname-c con 0x7fe1240a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:50.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1240b2310 0x7fe124133fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 1 -- 172.21.15.69:0/1933384203 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8032920 con 0x7fe4c80a8b20 2024-01-21T04:33:50.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:50.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 10 monclient: dump: 2024-01-21T04:33:50.780 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.780 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.780 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.780 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.780 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.780 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 1 -- 172.21.15.69:0/1933384203 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b8032aa0 con 0x7fe4c80a8b20 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 1 -- 172.21.15.69:0/1933384203 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8032e50 con 0x7fe4c80a8b20 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c4ff9640 10 monclient: dump: 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.781 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.782 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.782 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.782 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.782 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.782 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.782 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.782 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.782 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 10 monclient(hunting): picked mon.noname-b con 0x7fe1240b2310 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:50.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4c67fc640 4 set_mon_vals no callback set 2024-01-21T04:33:50.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4ce4de640 5 monclient: authenticate success, global_id 4335 2024-01-21T04:33:50.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1241387d0 con 0x7fe1240a8b20 2024-01-21T04:33:50.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4ce4de640 10 monclient: _renew_subs 2024-01-21T04:33:50.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4ce4de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe124134500 con 0x7fe1240b2310 2024-01-21T04:33:50.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe12c73e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:50.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.777+0000 7fe12c73e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe124139720 con 0x7fe1240a9d10 2024-01-21T04:33:50.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.777+0000 7fe12c73e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:50.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.777+0000 7fe12c73e640 10 monclient(hunting): authenticate will time out at 1389.118252s 2024-01-21T04:33:50.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.777+0000 7fe12a4da640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1240a8b20 0x7fe1241380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.776+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1933384203 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c8139b40 con 0x7fe4c80a8b20 2024-01-21T04:33:50.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.777+0000 7fe4ce4de640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:50.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.777+0000 7fe4ce4de640 10 monclient: _renew_subs 2024-01-21T04:33:50.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.777+0000 7fe4ce4de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.777+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1933384203 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c8138bd0 con 0x7fe4c80a8b20 2024-01-21T04:33:50.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.777+0000 7fe4ce4de640 1 librados: init done 2024-01-21T04:33:50.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.778+0000 7fe12a4da640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1240a8b20 0x7fe1241380d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53438/0 (socket says 172.21.15.69:53438) 2024-01-21T04:33:50.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.778+0000 7fe12a4da640 1 -- 172.21.15.69:0/3028786684 learned_addr learned my addr 172.21.15.69:0/3028786684 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:50.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.778+0000 7fe12a4da640 10 monclient(hunting): get_auth_request con 0x7fe1240a8b20 auth_method 0 2024-01-21T04:33:50.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.778+0000 7fe12a4da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:50.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.778+0000 7fe12a4da640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:50.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.778+0000 7fe12a4da640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:50.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.778+0000 7fe4ce4de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.779+0000 7fe4ce4de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.779+0000 7fe4ce4de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.779+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1933384203 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c80b1f80 con 0x7fe4c80a8b20 2024-01-21T04:33:50.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.786+0000 7fe12a4da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:50.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.786+0000 7fe12a4da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:50.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.786+0000 7fe12a4da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:50.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.789+0000 7fe12a4da640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-01-21T04:33:50.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.789+0000 7fe12a4da640 1 -- 172.21.15.69:0/3028786684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1240a9d10 msgr2=0x7fe124133a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.789+0000 7fe12a4da640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1240a9d10 0x7fe124133a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.789+0000 7fe12a4da640 1 -- 172.21.15.69:0/3028786684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1240b2310 msgr2=0x7fe124133fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:50.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.790+0000 7fe12a4da640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1240b2310 0x7fe124133fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:50.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.790+0000 7fe12a4da640 10 monclient: _finish_hunting 0 2024-01-21T04:33:50.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.790+0000 7fe12a4da640 1 monclient: found mon.noname-a 2024-01-21T04:33:50.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.790+0000 7fe12a4da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:50.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.790+0000 7fe12a4da640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:50.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.790+0000 7fe12a4da640 1 -- 172.21.15.69:0/3028786684 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c009a70 con 0x7fe1240a8b20 2024-01-21T04:33:50.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.813+0000 7fe4c4ff9640 1 -- 172.21.15.69:0/1933384203 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe4b804d3f0 con 0x7fe4c80a8b20 2024-01-21T04:33:50.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.814+0000 7fe4c4ff9640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe4b406c8b0 0x7fe4b406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.814+0000 7fe4c77fe640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe4b406c8b0 0x7fe4b406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.814+0000 7fe4c77fe640 10 monclient: get_auth_request con 0x7fe4b406c8b0 auth_method 0 2024-01-21T04:33:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.815+0000 7fe4c4ff9640 10 client.4335.objecter ms_handle_connect 0x7fe4b406c8b0 2024-01-21T04:33:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.815+0000 7fe4c77fe640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe4b406c8b0 0x7fe4b406ed70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc006fd0 tx=0x7fe4bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.832+0000 7fe4c4ff9640 1 -- 172.21.15.69:0/1933384203 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe4b807a8a0 con 0x7fe4c80a8b20 2024-01-21T04:33:50.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.832+0000 7fe4c4ff9640 10 client.4335.objecter ms_dispatch 0x7fe4c8000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:50.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.832+0000 7fe4c4ff9640 3 client.4335.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:50.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.832+0000 7fe4c4ff9640 3 client.4335.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:50.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.832+0000 7fe4c4ff9640 20 client.4335.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.833+0000 7fe12a4da640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1240a8b20 0x7fe1241380d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe12000e810 tx=0x7fe12000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:50.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.833+0000 7fe11affd640 10 client.?.objecter ms_handle_connect 0x7fe1240a8b20 2024-01-21T04:33:50.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.833+0000 7fe11affd640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:50.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.898+0000 7fe3f2e07640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T04:33:50.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.898+0000 7fe3f2e07640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T04:33:50.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.898+0000 7fe3f2e07640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:50.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.898+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe3ec132450 con 0x7fe3ec0a8b20 2024-01-21T04:33:50.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.906+0000 7f59aa9c0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T04:33:50.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.906+0000 7f59aa9c0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T04:33:50.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.906+0000 7f59aa9c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:50.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.906+0000 7f59aa9c0640 1 -- 172.21.15.69:0/4106471582 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f5984005600 con 0x7f59a40ac4e0 2024-01-21T04:33:50.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.978+0000 7fced97fa640 1 -- 172.21.15.69:0/3496662790 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fced0058910 con 0x7fced40c50e0 2024-01-21T04:33:50.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.978+0000 7fced97fa640 10 client.4351.objecter ms_dispatch 0x7fcedc000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:50.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.978+0000 7fced97fa640 3 client.4351.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:50.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.978+0000 7fced97fa640 3 client.4351.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:50.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.978+0000 7fced97fa640 20 client.4351.objecter dump_active .. 0 homeless 2024-01-21T04:33:50.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.990+0000 7fced97fa640 1 -- 172.21.15.69:0/3496662790 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fced0054550 con 0x7fced40c50e0 2024-01-21T04:33:50.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.990+0000 7fced97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:50.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.990+0000 7fced97fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:50.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 1 -- 172.21.15.69:0/3749889605 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68031870 con 0x7f8a780a8b20 2024-01-21T04:33:50.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:50.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 10 monclient: dump: 2024-01-21T04:33:50.997 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 10 monclient: _finish_auth 0 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 10 monclient: _check_auth_tickets 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:50.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 1 -- 172.21.15.69:0/3749889605 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a680319d0 con 0x7f8a780a8b20 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a7d93e640 5 monclient: authenticate success, global_id 4354 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 1 -- 172.21.15.69:0/3749889605 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68031d60 con 0x7f8a780a8b20 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a75ffb640 10 monclient: dump: 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:51.000 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a7d93e640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.996+0000 7f8a7d93e640 4 set_mon_vals no callback set 2024-01-21T04:33:51.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.997+0000 7f8a7d93e640 10 monclient: shutdown 2024-01-21T04:33:51.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.997+0000 7f8a7d93e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:51.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.997+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3749889605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a780a8b20 msgr2=0x7f8a78145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.997+0000 7f8a7d93e640 1 --2- 172.21.15.69:0/3749889605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a780a8b20 0x7f8a78145b80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8a780a8600 tx=0x7f8a6802b910 comp rx=0 tx=0).stop 2024-01-21T04:33:51.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.997+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3749889605 shutdown_connections 2024-01-21T04:33:51.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.997+0000 7f8a7d93e640 1 --2- 172.21.15.69:0/3749889605 >> v2:172.21.15.69:3300/0 conn(0x7f8a780b2310 0x7f8a7813f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.997+0000 7f8a7d93e640 1 --2- 172.21.15.69:0/3749889605 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a780a8b20 0x7f8a78145b80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.997+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3749889605 >> 172.21.15.69:0/3749889605 conn(0x7f8a78066fb0 msgr2=0x7f8a780b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.997+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3749889605 shutdown_connections 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.997+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3749889605 wait complete. 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.998+0000 7f8a7d93e640 10 monclient: build_initial_monmap 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.998+0000 7f8a7d93e640 10 monclient: monmap: 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.998+0000 7f8a7d93e640 1 librados: starting msgr at 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.998+0000 7f8a7d93e640 1 librados: starting objecter 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.998+0000 7f8a7d93e640 1 Processor -- start 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 1 -- start start 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 1 librados: setting wanted keys 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 1 librados: calling monclient init 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 10 monclient: init 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a780a8b20 0x7f8a7813be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 10 monclient(hunting): picked mon.noname-c con 0x7f8a780a8b20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a780a9d10 0x7f8a781394c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 10 monclient(hunting): picked mon.noname-b con 0x7f8a780a9d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:51.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a780b2310 0x7f8a78139a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:51.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 10 monclient(hunting): picked mon.noname-a con 0x7f8a780b2310 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:51.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:51.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a78141a70 con 0x7f8a780b2310 2024-01-21T04:33:51.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:51.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a78139f40 con 0x7f8a780a9d10 2024-01-21T04:33:51.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:51.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a78141cb0 con 0x7f8a780a8b20 2024-01-21T04:33:51.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:51.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:50.999+0000 7f8a7d93e640 10 monclient(hunting): authenticate will time out at 1389.341012s 2024-01-21T04:33:51.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.007+0000 7f59a0ff9640 1 -- 172.21.15.69:0/4106471582 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5998022020 con 0x7f59a40ac4e0 2024-01-21T04:33:51.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.007+0000 7f59a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T04:33:51.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.007+0000 7f59a0ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:51.008 INFO:teuthology.orchestra.run.smithi069.stdout:85899345927 2024-01-21T04:33:51.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 10 librados: watch_flush enter 2024-01-21T04:33:51.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 10 librados: watch_flush exit 2024-01-21T04:33:51.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 20 client.4348.objecter shutdown clearing up homeless session... 2024-01-21T04:33:51.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 10 client.4348.objecter successfully canceled tick 2024-01-21T04:33:51.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 -- 172.21.15.69:0/4106471582 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f599006c950 msgr2=0x7f599006ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f599006c950 0x7f599006ee10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5994033000 tx=0x7f599402f000 comp rx=0 tx=0).stop 2024-01-21T04:33:51.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f8a767fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a780a9d10 0x7f8a781394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 10 monclient: shutdown 2024-01-21T04:33:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 -- 172.21.15.69:0/4106471582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59a40ac4e0 msgr2=0x7f59a40acda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59a40ac4e0 0x7f59a40acda0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f599800b720 tx=0x7f599800bbe0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 -- 172.21.15.69:0/4106471582 shutdown_connections 2024-01-21T04:33:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f599006c950 0x7f599006ee10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59a412fe70 0x7f59a41302e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f59a40ac8c0 0x7f59a40ad2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 --2- 172.21.15.69:0/4106471582 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f59a40ac4e0 0x7f59a40acda0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 -- 172.21.15.69:0/4106471582 >> 172.21.15.69:0/4106471582 conn(0x7f59a4066fb0 msgr2=0x7f59a40b6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 -- 172.21.15.69:0/4106471582 shutdown_connections 2024-01-21T04:33:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 -- 172.21.15.69:0/4106471582 wait complete. 2024-01-21T04:33:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.009+0000 7f59aa9c0640 1 librados: shutdown 2024-01-21T04:33:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.011+0000 7f8a767fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a780a9d10 0x7f8a781394c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59894/0 (socket says 172.21.15.69:59894) 2024-01-21T04:33:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.011+0000 7f8a767fc640 1 -- 172.21.15.69:0/3968015171 learned_addr learned my addr 172.21.15.69:0/3968015171 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.011+0000 7f8a767fc640 10 monclient(hunting): get_auth_request con 0x7f8a780a9d10 auth_method 0 2024-01-21T04:33:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.011+0000 7f8a767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.011+0000 7f8a767fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.011+0000 7f8a767fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.012+0000 7f8a767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.012+0000 7f8a767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.012+0000 7f8a767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:51.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.016+0000 7f8a767fc640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2024-01-21T04:33:51.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.016+0000 7f8a767fc640 1 -- 172.21.15.69:0/3968015171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a780a8b20 msgr2=0x7f8a7813be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.016+0000 7f8a767fc640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a780a8b20 0x7f8a7813be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.016+0000 7f8a767fc640 1 -- 172.21.15.69:0/3968015171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a780b2310 msgr2=0x7f8a78139a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.016+0000 7f8a767fc640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a780b2310 0x7f8a78139a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.016+0000 7f8a767fc640 10 monclient: _finish_hunting 0 2024-01-21T04:33:51.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.016+0000 7f8a767fc640 1 monclient: found mon.noname-b 2024-01-21T04:33:51.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.016+0000 7f8a767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:51.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.016+0000 7f8a767fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:51.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.016+0000 7f8a767fc640 1 -- 172.21.15.69:0/3968015171 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a680094d0 con 0x7f8a780a9d10 2024-01-21T04:33:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.018+0000 7f8a767fc640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a780a9d10 0x7f8a781394c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00c400 tx=0x7f8a6c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.018+0000 7f8a5ffff640 10 client.?.objecter ms_handle_connect 0x7f8a780a9d10 2024-01-21T04:33:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.018+0000 7f8a5ffff640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:51.021 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345928 got 85899345927 for osd.4 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 1 -- 172.21.15.69:0/3968015171 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c0035f0 con 0x7f8a780a9d10 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 10 monclient: dump: 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 10 monclient: _finish_auth 0 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 10 monclient: _check_auth_tickets 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 1 -- 172.21.15.69:0/3968015171 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6c003750 con 0x7f8a780a9d10 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 1 -- 172.21.15.69:0/3968015171 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c003b40 con 0x7f8a780a9d10 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a7d93e640 5 monclient: authenticate success, global_id 4363 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a7d93e640 10 monclient: _renew_subs 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a7d93e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3968015171 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a78140280 con 0x7f8a780a9d10 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a5ffff640 10 monclient: dump: 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.021+0000 7f8a757fa640 4 set_mon_vals no callback set 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.022+0000 7f8a7d93e640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.022+0000 7f8a7d93e640 10 monclient: _renew_subs 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.022+0000 7f8a7d93e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:51.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.022+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3968015171 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a78140a20 con 0x7f8a780a9d10 2024-01-21T04:33:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.022+0000 7f8a7d93e640 1 librados: init done 2024-01-21T04:33:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.023+0000 7f8a7d93e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.023+0000 7f8a7d93e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.023+0000 7f8a7d93e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.023+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3968015171 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a781404a0 con 0x7f8a780a9d10 2024-01-21T04:33:51.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.043+0000 7f8a5ffff640 1 -- 172.21.15.69:0/3968015171 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8a6c003ca0 con 0x7f8a780a9d10 2024-01-21T04:33:51.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.043+0000 7f8a5ffff640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8a6406c860 0x7f8a6406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:51.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.044+0000 7f8a76ffd640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8a6406c860 0x7f8a6406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:51.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.044+0000 7f8a76ffd640 10 monclient: get_auth_request con 0x7f8a6406c860 auth_method 0 2024-01-21T04:33:51.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.044+0000 7f8a5ffff640 10 client.4363.objecter ms_handle_connect 0x7f8a6406c860 2024-01-21T04:33:51.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.044+0000 7f8a76ffd640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8a6406c860 0x7f8a6406ed20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8a68036000 tx=0x7f8a68030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.046+0000 7f8a5ffff640 1 -- 172.21.15.69:0/3968015171 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8a6c0010e0 con 0x7f8a780a9d10 2024-01-21T04:33:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.046+0000 7f8a5ffff640 10 client.4363.objecter ms_dispatch 0x7f8a78000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.046+0000 7f8a5ffff640 3 client.4363.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:51.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.046+0000 7f8a5ffff640 3 client.4363.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:51.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.046+0000 7f8a5ffff640 20 client.4363.objecter dump_active .. 0 homeless 2024-01-21T04:33:51.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.090+0000 7fe4c4ff9640 1 -- 172.21.15.69:0/1933384203 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe4b80764e0 con 0x7fe4c80a8b20 2024-01-21T04:33:51.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.091+0000 7fe4c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:51.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.091+0000 7fe4c4ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:51.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 1 -- 172.21.15.69:0/3028786684 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12000c8d0 con 0x7fe1240a8b20 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 10 monclient: dump: 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 10 monclient: _finish_auth 0 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:51.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 1 -- 172.21.15.69:0/3028786684 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12000ca30 con 0x7fe1240a8b20 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe128cd7640 4 set_mon_vals no callback set 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 1 -- 172.21.15.69:0/3028786684 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12000cde0 con 0x7fe1240a8b20 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe11affd640 10 monclient: dump: 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:51.098 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:51.099 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:51.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe12c73e640 5 monclient: authenticate success, global_id 4338 2024-01-21T04:33:51.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe12c73e640 10 monclient: _renew_subs 2024-01-21T04:33:51.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe12c73e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:51.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe12c73e640 1 -- 172.21.15.69:0/3028786684 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe124139ce0 con 0x7fe1240a8b20 2024-01-21T04:33:51.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe12c73e640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:51.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe12c73e640 10 monclient: _renew_subs 2024-01-21T04:33:51.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe12c73e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:51.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe12c73e640 1 -- 172.21.15.69:0/3028786684 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe124142ac0 con 0x7fe1240a8b20 2024-01-21T04:33:51.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.095+0000 7fe12c73e640 1 librados: init done 2024-01-21T04:33:51.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.096+0000 7fe12c73e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:51.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.096+0000 7fe12c73e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:51.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.096+0000 7fe12c73e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:51.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.096+0000 7fe12c73e640 1 -- 172.21.15.69:0/3028786684 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12404e970 con 0x7fe1240a8b20 2024-01-21T04:33:51.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.124+0000 7fa1418b4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T04:33:51.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.124+0000 7fa1418b4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T04:33:51.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.124+0000 7fa1418b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:51.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.124+0000 7fa1418b4640 1 -- 172.21.15.69:0/1250008644 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa13c06dca0 con 0x7fa13c0b0f60 2024-01-21T04:33:51.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.163+0000 7fe3e97fa640 1 -- 172.21.15.69:0/3704714150 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe3ec132450 con 0x7fe3ec0a8b20 2024-01-21T04:33:51.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.163+0000 7fe3e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T04:33:51.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.163+0000 7fe3e97fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:51.164 INFO:teuthology.orchestra.run.smithi069.stdout:55834574853 2024-01-21T04:33:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.165+0000 7fe3f2e07640 10 librados: watch_flush enter 2024-01-21T04:33:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.165+0000 7fe3f2e07640 10 librados: watch_flush exit 2024-01-21T04:33:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.165+0000 7fe3f2e07640 20 client.4289.objecter shutdown clearing up homeless session... 2024-01-21T04:33:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.165+0000 7fe3f2e07640 10 client.4289.objecter successfully canceled tick 2024-01-21T04:33:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.165+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe3d006c890 msgr2=0x7fe3d006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.165+0000 7fe3f2e07640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe3d006c890 0x7fe3d006ed50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec071d30 tx=0x7fe3d8009040 comp rx=0 tx=0).stop 2024-01-21T04:33:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.165+0000 7fe3f2e07640 10 monclient: shutdown 2024-01-21T04:33:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.165+0000 7fe3f2e07640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.165+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3ec0a8b20 msgr2=0x7fe3ec144df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.165+0000 7fe3f2e07640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3ec0a8b20 0x7fe3ec144df0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009890 tx=0x7fe3e00046c0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.166+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 shutdown_connections 2024-01-21T04:33:51.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.166+0000 7fe3f2e07640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe3d006c890 0x7fe3d006ed50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.166+0000 7fe3f2e07640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3ec0b2310 0x7fe3ec1e7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.166+0000 7fe3f2e07640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3ec0a9d10 0x7fe3ec142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.166+0000 7fe3f2e07640 1 --2- 172.21.15.69:0/3704714150 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe3ec0a8b20 0x7fe3ec144df0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.166+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 >> 172.21.15.69:0/3704714150 conn(0x7fe3ec066fb0 msgr2=0x7fe3ec0b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:51.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.166+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 shutdown_connections 2024-01-21T04:33:51.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.166+0000 7fe3f2e07640 1 -- 172.21.15.69:0/3704714150 wait complete. 2024-01-21T04:33:51.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.166+0000 7fe3f2e07640 1 librados: shutdown 2024-01-21T04:33:51.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-01-21T04:33:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.181+0000 7f5bd97fa640 1 -- 172.21.15.69:0/1612289380 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5bd4077080 con 0x7f5bdc137d60 2024-01-21T04:33:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.181+0000 7f5bd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.181+0000 7f5bd97fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:51.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.218+0000 7fcee1ea3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T04:33:51.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.218+0000 7fcee1ea3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T04:33:51.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.218+0000 7fcee1ea3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:51.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.218+0000 7fcee1ea3640 1 -- 172.21.15.69:0/3496662790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fced40058f0 con 0x7fced40c50e0 2024-01-21T04:33:51.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.286+0000 7f8a5ffff640 1 -- 172.21.15.69:0/3968015171 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8a6c053f90 con 0x7f8a780a9d10 2024-01-21T04:33:51.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.286+0000 7f8a5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:51.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.286+0000 7f8a5ffff640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:51.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.292+0000 7fced97fa640 1 -- 172.21.15.69:0/3496662790 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fced001e090 con 0x7fced40c50e0 2024-01-21T04:33:51.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.292+0000 7fced97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T04:33:51.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.292+0000 7fced97fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:51.293 INFO:teuthology.orchestra.run.smithi069.stdout:64424509444 2024-01-21T04:33:51.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 10 librados: watch_flush enter 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 10 librados: watch_flush exit 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 20 client.4351.objecter shutdown clearing up homeless session... 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 10 client.4351.objecter successfully canceled tick 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 -- 172.21.15.69:0/3496662790 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcec006c8b0 msgr2=0x7fcec006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcec006c8b0 0x7fcec006ed70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcec8038000 tx=0x7fcec8030000 comp rx=0 tx=0).stop 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 10 monclient: shutdown 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 -- 172.21.15.69:0/3496662790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fced40c50e0 msgr2=0x7fced402dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fced40c50e0 0x7fced402dce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fced000b4a0 tx=0x7fced000b960 comp rx=0 tx=0).stop 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 -- 172.21.15.69:0/3496662790 shutdown_connections 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fcec006c8b0 0x7fcec006ed70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fced415e930 0x7fced415edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fced40c50e0 0x7fced402dce0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 --2- 172.21.15.69:0/3496662790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fced402b740 0x7fced40c6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 -- 172.21.15.69:0/3496662790 >> 172.21.15.69:0/3496662790 conn(0x7fced403a920 msgr2=0x7fced40be140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:51.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 -- 172.21.15.69:0/3496662790 shutdown_connections 2024-01-21T04:33:51.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 -- 172.21.15.69:0/3496662790 wait complete. 2024-01-21T04:33:51.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.295+0000 7fcee1ea3640 1 librados: shutdown 2024-01-21T04:33:51.312 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.0 2024-01-21T04:33:51.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.317+0000 7fe4ce4de640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T04:33:51.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.317+0000 7fe4ce4de640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T04:33:51.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.317+0000 7fe4ce4de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:51.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.317+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1933384203 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fe4c804e970 con 0x7fe4c80a8b20 2024-01-21T04:33:51.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.360+0000 7f5be2a90640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T04:33:51.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.360+0000 7f5be2a90640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T04:33:51.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.360+0000 7f5be2a90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:51.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.360+0000 7f5be2a90640 1 -- 172.21.15.69:0/1612289380 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5bdc04e970 con 0x7f5bdc137d60 2024-01-21T04:33:51.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.363+0000 7fe11affd640 1 -- 172.21.15.69:0/3028786684 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fe120026430 con 0x7fe1240a8b20 2024-01-21T04:33:51.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.364+0000 7fe11affd640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10806c8b0 0x7fe10806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:51.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.364+0000 7fe129cd9640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10806c8b0 0x7fe10806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:51.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.364+0000 7fe129cd9640 10 monclient: get_auth_request con 0x7fe10806c8b0 auth_method 0 2024-01-21T04:33:51.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.365+0000 7fe11affd640 10 client.4338.objecter ms_handle_connect 0x7fe10806c8b0 2024-01-21T04:33:51.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.365+0000 7fe129cd9640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10806c8b0 0x7fe10806ed70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe114000f60 tx=0x7fe1140079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:51.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.366+0000 7f5bd97fa640 1 -- 172.21.15.69:0/1612289380 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5bd407a6a0 con 0x7f5bdc137d60 2024-01-21T04:33:51.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.366+0000 7f5bd97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T04:33:51.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.366+0000 7f5bd97fa640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:51.367 INFO:teuthology.orchestra.run.smithi069.stdout:85899345927 2024-01-21T04:33:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 10 librados: watch_flush enter 2024-01-21T04:33:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 10 librados: watch_flush exit 2024-01-21T04:33:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 20 client.4295.objecter shutdown clearing up homeless session... 2024-01-21T04:33:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 10 client.4295.objecter successfully canceled tick 2024-01-21T04:33:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 1 -- 172.21.15.69:0/1612289380 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5bc806c890 msgr2=0x7f5bc806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5bc806c890 0x7f5bc806ed50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0014e0 tx=0x7f5bcc007990 comp rx=0 tx=0).stop 2024-01-21T04:33:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 10 monclient: shutdown 2024-01-21T04:33:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 1 -- 172.21.15.69:0/1612289380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bdc137d60 msgr2=0x7f5bdc1381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bdc137d60 0x7f5bdc1381e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009f30 tx=0x7f5bd4004b50 comp rx=0 tx=0).stop 2024-01-21T04:33:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 1 -- 172.21.15.69:0/1612289380 shutdown_connections 2024-01-21T04:33:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5bc806c890 0x7f5bc806ed50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5bdc137d60 0x7f5bdc1381e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bdc1d7470 0x7f5bdc1d78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 1 --2- 172.21.15.69:0/1612289380 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5bc40052a0 0x7f5bdc0abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.368+0000 7f5be2a90640 1 -- 172.21.15.69:0/1612289380 >> 172.21.15.69:0/1612289380 conn(0x7f5bdc066fb0 msgr2=0x7f5bdc13cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.369+0000 7f5be2a90640 1 -- 172.21.15.69:0/1612289380 shutdown_connections 2024-01-21T04:33:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.369+0000 7f5be2a90640 1 -- 172.21.15.69:0/1612289380 wait complete. 2024-01-21T04:33:51.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.369+0000 7f5be2a90640 1 librados: shutdown 2024-01-21T04:33:51.380 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345928 got 85899345927 for osd.6 2024-01-21T04:33:51.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.392+0000 7fe11affd640 1 -- 172.21.15.69:0/3028786684 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe120058b60 con 0x7fe1240a8b20 2024-01-21T04:33:51.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.392+0000 7fe11affd640 10 client.4338.objecter ms_dispatch 0x7fe124000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:51.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.392+0000 7fe11affd640 3 client.4338.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:51.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.392+0000 7fe11affd640 3 client.4338.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:51.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.392+0000 7fe11affd640 20 client.4338.objecter dump_active .. 0 homeless 2024-01-21T04:33:51.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.392+0000 7fe11affd640 1 -- 172.21.15.69:0/3028786684 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fe1200547a0 con 0x7fe1240a8b20 2024-01-21T04:33:51.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.392+0000 7fe11affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:51.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.392+0000 7fe11affd640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:51.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.432+0000 7fa138ff9640 1 -- 172.21.15.69:0/1250008644 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa12c01e070 con 0x7fa13c0b0f60 2024-01-21T04:33:51.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.432+0000 7fa138ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T04:33:51.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.432+0000 7fa138ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:51.433 INFO:teuthology.orchestra.run.smithi069.stdout:64424509445 2024-01-21T04:33:51.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.434+0000 7fa1418b4640 10 librados: watch_flush enter 2024-01-21T04:33:51.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.434+0000 7fa1418b4640 10 librados: watch_flush exit 2024-01-21T04:33:51.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.434+0000 7fa1418b4640 20 client.4326.objecter shutdown clearing up homeless session... 2024-01-21T04:33:51.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.434+0000 7fa1418b4640 10 client.4326.objecter successfully canceled tick 2024-01-21T04:33:51.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.434+0000 7fa1418b4640 1 -- 172.21.15.69:0/1250008644 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa11c06c8e0 msgr2=0x7fa11c06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.434+0000 7fa1418b4640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa11c06c8e0 0x7fa11c06eda0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa124036000 tx=0x7fa124030000 comp rx=0 tx=0).stop 2024-01-21T04:33:51.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.434+0000 7fa1418b4640 10 monclient: shutdown 2024-01-21T04:33:51.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.434+0000 7fa1418b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:51.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.434+0000 7fa1418b4640 1 -- 172.21.15.69:0/1250008644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa13c0b0f60 msgr2=0x7fa13c14a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.434+0000 7fa1418b4640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa13c0b0f60 0x7fa13c14a0a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa12c00c820 tx=0x7fa12c00cce0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.435+0000 7fa1418b4640 1 -- 172.21.15.69:0/1250008644 shutdown_connections 2024-01-21T04:33:51.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.435+0000 7fa1418b4640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa11c06c8e0 0x7fa11c06eda0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.435+0000 7fa1418b4640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa13c1469b0 0x7fa13c141250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.435+0000 7fa1418b4640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa13c0b3890 0x7fa13c140d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.435+0000 7fa1418b4640 1 --2- 172.21.15.69:0/1250008644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa13c0b0f60 0x7fa13c14a0a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.435+0000 7fa1418b4640 1 -- 172.21.15.69:0/1250008644 >> 172.21.15.69:0/1250008644 conn(0x7fa13c066fb0 msgr2=0x7fa13c145360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:51.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.435+0000 7fa1418b4640 1 -- 172.21.15.69:0/1250008644 shutdown_connections 2024-01-21T04:33:51.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.435+0000 7fa1418b4640 1 -- 172.21.15.69:0/1250008644 wait complete. 2024-01-21T04:33:51.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.435+0000 7fa1418b4640 1 librados: shutdown 2024-01-21T04:33:51.450 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.1 2024-01-21T04:33:51.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.467+0000 7fe4c4ff9640 1 -- 172.21.15.69:0/1933384203 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe4b8079b00 con 0x7fe4c80a8b20 2024-01-21T04:33:51.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.467+0000 7fe4c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T04:33:51.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.467+0000 7fe4c4ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:51.468 INFO:teuthology.orchestra.run.smithi069.stdout:64424509446 2024-01-21T04:33:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 10 librados: watch_flush enter 2024-01-21T04:33:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 10 librados: watch_flush exit 2024-01-21T04:33:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 20 client.4335.objecter shutdown clearing up homeless session... 2024-01-21T04:33:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 10 client.4335.objecter successfully canceled tick 2024-01-21T04:33:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1933384203 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe4b406c8b0 msgr2=0x7fe4b406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe4b406c8b0 0x7fe4b406ed70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc006fd0 tx=0x7fe4bc008040 comp rx=0 tx=0).stop 2024-01-21T04:33:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 10 monclient: shutdown 2024-01-21T04:33:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1933384203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4c80a8b20 msgr2=0x7fe4c8142930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4c80a8b20 0x7fe4c8142930 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8033000 tx=0x7fe4b80046c0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1933384203 shutdown_connections 2024-01-21T04:33:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe4b406c8b0 0x7fe4b406ed70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe4c80b2310 0x7fe4c813a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4c80a9d10 0x7fe4c8138040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 1 --2- 172.21.15.69:0/1933384203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4c80a8b20 0x7fe4c8142930 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.470+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1933384203 >> 172.21.15.69:0/1933384203 conn(0x7fe4c8066fb0 msgr2=0x7fe4c813ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.471+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1933384203 shutdown_connections 2024-01-21T04:33:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.471+0000 7fe4ce4de640 1 -- 172.21.15.69:0/1933384203 wait complete. 2024-01-21T04:33:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.471+0000 7fe4ce4de640 1 librados: shutdown 2024-01-21T04:33:51.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509446 for osd.7 2024-01-21T04:33:51.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.489+0000 7f8a7d93e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T04:33:51.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.490+0000 7f8a7d93e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T04:33:51.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.490+0000 7f8a7d93e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:51.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.490+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3968015171 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8a7804e970 con 0x7f8a780a9d10 2024-01-21T04:33:51.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.537+0000 7f8a5ffff640 1 -- 172.21.15.69:0/3968015171 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8a6c020090 con 0x7f8a780a9d10 2024-01-21T04:33:51.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.537+0000 7f8a5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T04:33:51.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.537+0000 7f8a5ffff640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:51.538 INFO:teuthology.orchestra.run.smithi069.stdout:64424509447 2024-01-21T04:33:51.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 10 librados: watch_flush enter 2024-01-21T04:33:51.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 10 librados: watch_flush exit 2024-01-21T04:33:51.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 20 client.4363.objecter shutdown clearing up homeless session... 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 10 client.4363.objecter successfully canceled tick 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3968015171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8a6406c860 msgr2=0x7f8a6406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8a6406c860 0x7f8a6406ed20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8a68036000 tx=0x7f8a68030000 comp rx=0 tx=0).stop 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 10 monclient: shutdown 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3968015171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a780a9d10 msgr2=0x7f8a781394c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a780a9d10 0x7f8a781394c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c00c400 tx=0x7f8a6c00c8c0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3968015171 shutdown_connections 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8a6406c860 0x7f8a6406ed20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a780b2310 0x7f8a78139a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a780a9d10 0x7f8a781394c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 --2- 172.21.15.69:0/3968015171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8a780a8b20 0x7f8a7813be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3968015171 >> 172.21.15.69:0/3968015171 conn(0x7f8a78066fb0 msgr2=0x7f8a780b6490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:51.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3968015171 shutdown_connections 2024-01-21T04:33:51.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 -- 172.21.15.69:0/3968015171 wait complete. 2024-01-21T04:33:51.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.540+0000 7f8a7d93e640 1 librados: shutdown 2024-01-21T04:33:51.555 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.2 2024-01-21T04:33:51.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.612+0000 7fe12c73e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T04:33:51.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.612+0000 7fe12c73e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T04:33:51.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.612+0000 7fe12c73e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:51.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.612+0000 7fe12c73e640 1 -- 172.21.15.69:0/3028786684 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe1240b4bb0 con 0x7fe1240a8b20 2024-01-21T04:33:51.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.633+0000 7fe11affd640 1 -- 172.21.15.69:0/3028786684 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe120021090 con 0x7fe1240a8b20 2024-01-21T04:33:51.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.633+0000 7fe11affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T04:33:51.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.633+0000 7fe11affd640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:51.635 INFO:teuthology.orchestra.run.smithi069.stdout:90194313225 2024-01-21T04:33:51.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 10 librados: watch_flush enter 2024-01-21T04:33:51.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 10 librados: watch_flush exit 2024-01-21T04:33:51.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 20 client.4338.objecter shutdown clearing up homeless session... 2024-01-21T04:33:51.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 10 client.4338.objecter successfully canceled tick 2024-01-21T04:33:51.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 1 -- 172.21.15.69:0/3028786684 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10806c8b0 msgr2=0x7fe10806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10806c8b0 0x7fe10806ed70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe114000f60 tx=0x7fe1140079d0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 10 monclient: shutdown 2024-01-21T04:33:51.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:51.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 1 -- 172.21.15.69:0/3028786684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1240a8b20 msgr2=0x7fe1241380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:51.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1240a8b20 0x7fe1241380d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe12000e810 tx=0x7fe12000ecd0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 1 -- 172.21.15.69:0/3028786684 shutdown_connections 2024-01-21T04:33:51.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fe10806c8b0 0x7fe10806ed70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1240b2310 0x7fe124133fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fe1240a9d10 0x7fe124133a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 1 --2- 172.21.15.69:0/3028786684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1240a8b20 0x7fe1241380d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:51.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.635+0000 7fe12c73e640 1 -- 172.21.15.69:0/3028786684 >> 172.21.15.69:0/3028786684 conn(0x7fe124066fb0 msgr2=0x7fe1240b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:51.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.636+0000 7fe12c73e640 1 -- 172.21.15.69:0/3028786684 shutdown_connections 2024-01-21T04:33:51.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.636+0000 7fe12c73e640 1 -- 172.21.15.69:0/3028786684 wait complete. 2024-01-21T04:33:51.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:51.636+0000 7fe12c73e640 1 librados: shutdown 2024-01-21T04:33:51.648 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313226 got 90194313225 for osd.3 2024-01-21T04:33:52.022 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-21T04:33:52.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.121+0000 7f8c09415640 10 monclient: get_monmap_and_config 2024-01-21T04:33:52.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.121+0000 7f8c09415640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.121+0000 7f8c09415640 10 monclient: monmap: 2024-01-21T04:33:52.122 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.123 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.123 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:52.122649+0000 2024-01-21T04:33:52.123 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:52.122649+0000 2024-01-21T04:33:52.123 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.123 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.123 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:52.123 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:52.123 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:52.123 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:52.123 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:52.123 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.122+0000 7f8c09415640 1 Processor -- start 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.122+0000 7f8c09415640 1 -- start start 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.122+0000 7f8c09415640 10 monclient: init 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.122+0000 7f8c09415640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.122+0000 7f8c09415640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.122+0000 7f8c09415640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8c040a8ab0 0x7f8c041426b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.122+0000 7f8c09415640 10 monclient(hunting): picked mon.noname-c con 0x7f8c040a8ab0 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c09415640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c04142bf0 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c09415640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8c0413f950 0x7f8c0414c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c09415640 10 monclient(hunting): picked mon.noname-a con 0x7f8c0413f950 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c09415640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c040aaa20 con 0x7f8c04142bf0 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c09415640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c09415640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c040aa820 con 0x7f8c0413f950 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c09415640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c09415640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c040aad00 con 0x7f8c040a8ab0 2024-01-21T04:33:52.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c09415640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c09415640 10 monclient(hunting): authenticate will time out at 1390.464270s 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c037fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8c0413f950 0x7f8c0414c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.123+0000 7f8c02ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f8c040a8ab0 0x7f8c041426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c037fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8c0413f950 0x7f8c0414c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53464/0 (socket says 172.21.15.69:53464) 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c037fe640 1 -- 172.21.15.69:0/2734184014 learned_addr learned my addr 172.21.15.69:0/2734184014 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c037fe640 10 monclient(hunting): get_auth_request con 0x7f8c0413f950 auth_method 0 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c037fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c037fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c02ffd640 10 monclient(hunting): get_auth_request con 0x7f8c040a8ab0 auth_method 0 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c02ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.124+0000 7f8c037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.125+0000 7f8c02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.125+0000 7f8c02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.125+0000 7f8c02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.126+0000 7f8c037fe640 10 monclient(hunting): handle_auth_done global_id 4341 payload 274 2024-01-21T04:33:52.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.126+0000 7f8c037fe640 1 -- 172.21.15.69:0/2734184014 >> v2:172.21.15.144:3300/0 conn(0x7f8c040a8ab0 msgr2=0x7f8c041426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.126+0000 7f8c037fe640 1 --2- 172.21.15.69:0/2734184014 >> v2:172.21.15.144:3300/0 conn(0x7f8c040a8ab0 0x7f8c041426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.126+0000 7f8c037fe640 1 -- 172.21.15.69:0/2734184014 >> v1:172.21.15.144:6789/0 conn(0x7f8c04142bf0 legacy=0x7f8c0413f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.126+0000 7f8c037fe640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.126+0000 7f8c037fe640 1 monclient: found mon.noname-a 2024-01-21T04:33:52.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.126+0000 7f8c037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.126+0000 7f8c037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.126+0000 7f8c037fe640 1 -- 172.21.15.69:0/2734184014 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c040aafe0 con 0x7f8c0413f950 2024-01-21T04:33:52.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.128+0000 7f8c037fe640 1 --2- 172.21.15.69:0/2734184014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0413f950 0x7f8c0414c900 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8c040b89b0 tx=0x7f8bfc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb5a8d76fe949d5c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 1 -- 172.21.15.69:0/2734184014 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc02ca80 con 0x7f8c0413f950 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 10 monclient: dump: 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.134 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c09415640 5 monclient: authenticate success, global_id 4341 2024-01-21T04:33:52.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 1 -- 172.21.15.69:0/2734184014 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bfc02cbe0 con 0x7f8c0413f950 2024-01-21T04:33:52.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c09415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:52.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c09415640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:52.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 1 -- 172.21.15.69:0/2734184014 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc0343f0 con 0x7f8c0413f950 2024-01-21T04:33:52.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c09415640 4 set_mon_vals no callback set 2024-01-21T04:33:52.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.132+0000 7f8c027fc640 10 monclient: dump: 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.136 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.133+0000 7f8c09415640 10 monclient: shutdown 2024-01-21T04:33:52.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.133+0000 7f8c09415640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:52.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.133+0000 7f8c09415640 1 -- 172.21.15.69:0/2734184014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0413f950 msgr2=0x7f8c0414c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.133+0000 7f8c09415640 1 --2- 172.21.15.69:0/2734184014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0413f950 0x7f8c0414c900 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8c040b89b0 tx=0x7f8bfc02bee0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.133+0000 7f8c09415640 1 -- 172.21.15.69:0/2734184014 shutdown_connections 2024-01-21T04:33:52.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.133+0000 7f8c09415640 1 --2- 172.21.15.69:0/2734184014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0413f950 0x7f8c0414c900 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.133+0000 7f8c09415640 1 --2- 172.21.15.69:0/2734184014 >> v2:172.21.15.144:3300/0 conn(0x7f8c040a8ab0 0x7f8c041426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.133+0000 7f8c09415640 1 -- 172.21.15.69:0/2734184014 >> 172.21.15.69:0/2734184014 conn(0x7f8c04066f00 msgr2=0x7f8c040b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.133+0000 7f8c09415640 1 -- 172.21.15.69:0/2734184014 shutdown_connections 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.133+0000 7f8c09415640 1 -- 172.21.15.69:0/2734184014 wait complete. 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.134+0000 7f8c09415640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.134+0000 7f8c09415640 10 monclient: monmap: 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:52.141 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.134+0000 7f8c09415640 1 librados: starting msgr at 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.134+0000 7f8c09415640 1 librados: starting objecter 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.134+0000 7f8c09415640 1 Processor -- start 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 1 -- start start 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 1 librados: setting wanted keys 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 1 librados: calling monclient init 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 10 monclient: init 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c040a8ab0 0x7f8c04143e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 10 monclient(hunting): picked mon.noname-c con 0x7f8c040a8ab0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0413f950 0x7f8c041449e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 10 monclient(hunting): picked mon.noname-a con 0x7f8c0413f950 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c04144f20 0x7f8c041d7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 10 monclient(hunting): picked mon.noname-b con 0x7f8c04144f20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c041d7340 con 0x7f8c0413f950 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c04145c40 con 0x7f8c04144f20 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c04145e90 con 0x7f8c040a8ab0 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.135+0000 7f8c09415640 10 monclient(hunting): authenticate will time out at 1390.476871s 2024-01-21T04:33:52.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8bfbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0413f950 0x7f8c041449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8c037fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c04144f20 0x7f8c041d7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8c02ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c040a8ab0 0x7f8c04143e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8bfbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0413f950 0x7f8c041449e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53480/0 (socket says 172.21.15.69:53480) 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8bfbfff640 1 -- 172.21.15.69:0/3279896228 learned_addr learned my addr 172.21.15.69:0/3279896228 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8bfbfff640 10 monclient(hunting): get_auth_request con 0x7f8c0413f950 auth_method 0 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8bfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8bfbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8bfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8c037fe640 10 monclient(hunting): get_auth_request con 0x7f8c04144f20 auth_method 0 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8c037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8c037fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8c037fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8c02ffd640 10 monclient(hunting): get_auth_request con 0x7f8c040a8ab0 auth_method 0 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8c02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8c02ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.136+0000 7f8c02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.137+0000 7f8bfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.137+0000 7f8bfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.137+0000 7f8bfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.137+0000 7f8c037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.137+0000 7f8c037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.137+0000 7f8c037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.137+0000 7f8c02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.137+0000 7f8c02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.137+0000 7f8c02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.139+0000 7f8c037fe640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2024-01-21T04:33:52.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.139+0000 7f8c037fe640 1 -- 172.21.15.69:0/3279896228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c040a8ab0 msgr2=0x7f8c04143e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.140+0000 7f8c037fe640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c040a8ab0 0x7f8c04143e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.140+0000 7f8c037fe640 1 -- 172.21.15.69:0/3279896228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0413f950 msgr2=0x7f8c041449e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.140+0000 7f8c037fe640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0413f950 0x7f8c041449e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.140+0000 7f8c037fe640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.140+0000 7f8c037fe640 1 monclient: found mon.noname-b 2024-01-21T04:33:52.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.140+0000 7f8c037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.140+0000 7f8c037fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.140+0000 7f8c037fe640 1 -- 172.21.15.69:0/3279896228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc009b30 con 0x7f8c04144f20 2024-01-21T04:33:52.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.142+0000 7f8c037fe640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c04144f20 0x7f8c041d7760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc032000 tx=0x7f8bfc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.142+0000 7f8bfb7fe640 10 client.?.objecter ms_handle_connect 0x7f8c04144f20 2024-01-21T04:33:52.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.142+0000 7f8bfb7fe640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:52.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8bfb7fe640 1 -- 172.21.15.69:0/3279896228 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc009510 con 0x7f8c04144f20 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8bfb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8bfb7fe640 10 monclient: dump: 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8bfb7fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8bfb7fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8bfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8bfb7fe640 1 -- 172.21.15.69:0/3279896228 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bfc03e450 con 0x7f8c04144f20 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8bfb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8bfb7fe640 1 -- 172.21.15.69:0/3279896228 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc03e800 con 0x7f8c04144f20 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8c09415640 5 monclient: authenticate success, global_id 4369 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8c09415640 10 monclient: _renew_subs 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8c09415640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8c09415640 1 -- 172.21.15.69:0/3279896228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0413ccd0 con 0x7f8c04144f20 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8c01ffb640 4 set_mon_vals no callback set 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.146+0000 7f8bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.147+0000 7f8bfb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:52.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.147+0000 7f8bfb7fe640 10 monclient: dump: 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.147+0000 7f8c09415640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.147+0000 7f8c09415640 10 monclient: _renew_subs 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.147+0000 7f8c09415640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.147+0000 7f8c09415640 1 -- 172.21.15.69:0/3279896228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c041d7e80 con 0x7f8c04144f20 2024-01-21T04:33:52.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.147+0000 7f8c09415640 1 librados: init done 2024-01-21T04:33:52.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.147+0000 7f8c09415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.147+0000 7f8c09415640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.147+0000 7f8c09415640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.147+0000 7f8c09415640 1 -- 172.21.15.69:0/3279896228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c041d7480 con 0x7f8c04144f20 2024-01-21T04:33:52.182 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-21T04:33:52.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.281+0000 7fdd4877d640 10 monclient: get_monmap_and_config 2024-01-21T04:33:52.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.281+0000 7fdd4877d640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.281+0000 7fdd4877d640 10 monclient: monmap: 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:52.282858+0000 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:52.282858+0000 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:52.283 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.282+0000 7fdd4877d640 1 Processor -- start 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.282+0000 7fdd4877d640 1 -- start start 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.282+0000 7fdd4877d640 10 monclient: init 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd400a8750 0x7fdd40070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 10 monclient(hunting): picked mon.noname-a con 0x7fdd400a8750 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdd400720e0 0x7fdd40142270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 10 monclient(hunting): picked mon.noname-c con 0x7fdd400720e0 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd4013f520 0x7fdd4013f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 10 monclient(hunting): picked mon.noname-b con 0x7fdd4013f520 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd40071500 con 0x7fdd400a8750 2024-01-21T04:33:52.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd400a8b30 con 0x7fdd400720e0 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4013fed0 con 0x7fdd4013f520 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd4877d640 10 monclient(hunting): authenticate will time out at 1390.624634s 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd46519640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd400a8750 0x7fdd40070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.283+0000 7fdd46d1a640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd4013f520 0x7fdd4013f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46519640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd400a8750 0x7fdd40070580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53486/0 (socket says 172.21.15.69:53486) 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46519640 1 -- 172.21.15.69:0/1912939186 learned_addr learned my addr 172.21.15.69:0/1912939186 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46519640 10 monclient(hunting): get_auth_request con 0x7fdd400a8750 auth_method 0 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46519640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46519640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46519640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46d1a640 10 monclient(hunting): get_auth_request con 0x7fdd4013f520 auth_method 0 2024-01-21T04:33:52.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46d1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46d1a640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46d1a640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46519640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46519640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46519640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46d1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46d1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.284+0000 7fdd46d1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.286+0000 7fdd46d1a640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-01-21T04:33:52.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.286+0000 7fdd46d1a640 1 -- 172.21.15.69:0/1912939186 >> v2:172.21.15.144:3300/0 conn(0x7fdd400720e0 msgr2=0x7fdd40142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.286+0000 7fdd46d1a640 1 --2- 172.21.15.69:0/1912939186 >> v2:172.21.15.144:3300/0 conn(0x7fdd400720e0 0x7fdd40142270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.286+0000 7fdd46d1a640 1 -- 172.21.15.69:0/1912939186 >> v2:172.21.15.69:3300/0 conn(0x7fdd400a8750 msgr2=0x7fdd40070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.286+0000 7fdd46d1a640 1 --2- 172.21.15.69:0/1912939186 >> v2:172.21.15.69:3300/0 conn(0x7fdd400a8750 0x7fdd40070580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.286+0000 7fdd46d1a640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.286+0000 7fdd46d1a640 1 monclient: found mon.noname-b 2024-01-21T04:33:52.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.286+0000 7fdd46d1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.286+0000 7fdd46d1a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:33:52.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.286+0000 7fdd46d1a640 1 -- 172.21.15.69:0/1912939186 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd400b1410 con 0x7fdd4013f520 2024-01-21T04:33:52.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.286+0000 7fdd46519640 1 --2- 172.21.15.69:0/1912939186 >> v2:172.21.15.69:3300/0 conn(0x7fdd400a8750 0x7fdd40070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T04:33:52.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.288+0000 7fdd46d1a640 1 --2- 172.21.15.69:0/1912939186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd4013f520 0x7fdd4013f990 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdd400acc90 tx=0x7fdd3802bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=300f18c31e09a5f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 1 -- 172.21.15.69:0/1912939186 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3802cb30 con 0x7fdd4013f520 2024-01-21T04:33:52.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 10 monclient: dump: 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 1 -- 172.21.15.69:0/1912939186 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3802cc90 con 0x7fdd4013f520 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 1 -- 172.21.15.69:0/1912939186 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38034440 con 0x7fdd4013f520 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd4877d640 5 monclient: authenticate success, global_id 4301 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd4877d640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd4877d640 4 set_mon_vals no callback set 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.291+0000 7fdd45517640 10 monclient: dump: 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.294 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.295 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.295 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.295 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.295 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.295 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.295 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.292+0000 7fdd4877d640 10 monclient: shutdown 2024-01-21T04:33:52.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.292+0000 7fdd4877d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:52.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.292+0000 7fdd4877d640 1 -- 172.21.15.69:0/1912939186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd4013f520 msgr2=0x7fdd4013f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.292+0000 7fdd4877d640 1 --2- 172.21.15.69:0/1912939186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd4013f520 0x7fdd4013f990 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdd400acc90 tx=0x7fdd3802bd80 comp rx=0 tx=0).stop 2024-01-21T04:33:52.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.292+0000 7fdd4877d640 1 -- 172.21.15.69:0/1912939186 shutdown_connections 2024-01-21T04:33:52.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.292+0000 7fdd4877d640 1 --2- 172.21.15.69:0/1912939186 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd4013f520 0x7fdd4013f990 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.292+0000 7fdd4877d640 1 --2- 172.21.15.69:0/1912939186 >> v2:172.21.15.144:3300/0 conn(0x7fdd400720e0 0x7fdd40142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.292+0000 7fdd4877d640 1 --2- 172.21.15.69:0/1912939186 >> v2:172.21.15.69:3300/0 conn(0x7fdd400a8750 0x7fdd40070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.292+0000 7fdd4877d640 1 -- 172.21.15.69:0/1912939186 >> 172.21.15.69:0/1912939186 conn(0x7fdd40066f00 msgr2=0x7fdd400b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.292+0000 7fdd4877d640 1 -- 172.21.15.69:0/1912939186 shutdown_connections 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.292+0000 7fdd4877d640 1 -- 172.21.15.69:0/1912939186 wait complete. 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.293+0000 7fdd4877d640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.293+0000 7fdd4877d640 10 monclient: monmap: 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.293+0000 7fdd4877d640 1 librados: starting msgr at 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.293+0000 7fdd4877d640 1 librados: starting objecter 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.293+0000 7fdd4877d640 1 Processor -- start 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.293+0000 7fdd4877d640 1 -- start start 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.293+0000 7fdd4877d640 1 librados: setting wanted keys 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.293+0000 7fdd4877d640 1 librados: calling monclient init 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.293+0000 7fdd4877d640 10 monclient: init 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd400720e0 0x7fdd40141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 10 monclient(hunting): picked mon.noname-a con 0x7fdd400720e0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd400a8750 0x7fdd400ab630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 10 monclient(hunting): picked mon.noname-c con 0x7fdd400a8750 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd4013f520 0x7fdd400a97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 10 monclient(hunting): picked mon.noname-b con 0x7fdd4013f520 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd400ac060 con 0x7fdd400720e0 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd400aade0 con 0x7fdd4013f520 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd400aaf50 con 0x7fdd400a8750 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd4877d640 10 monclient(hunting): authenticate will time out at 1390.635559s 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd46519640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd400720e0 0x7fdd40141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.294+0000 7fdd45d18640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd400a8750 0x7fdd400ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd45d18640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd400a8750 0x7fdd400ab630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:46996/0 (socket says 172.21.15.69:46996) 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd46519640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd400720e0 0x7fdd40141690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53500/0 (socket says 172.21.15.69:53500) 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd45d18640 1 -- 172.21.15.69:0/3672666549 learned_addr learned my addr 172.21.15.69:0/3672666549 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd45d18640 10 monclient(hunting): get_auth_request con 0x7fdd400a8750 auth_method 0 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd45d18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd45d18640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd45d18640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd46519640 10 monclient(hunting): get_auth_request con 0x7fdd400720e0 auth_method 0 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd46519640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd46519640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd46519640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd45d18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd45d18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd45d18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd46519640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd46519640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.295+0000 7fdd46519640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.298+0000 7fdd46519640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-01-21T04:33:52.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.298+0000 7fdd46519640 1 -- 172.21.15.69:0/3672666549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd400a8750 msgr2=0x7fdd400ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.298+0000 7fdd46519640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd400a8750 0x7fdd400ab630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.298+0000 7fdd46519640 1 -- 172.21.15.69:0/3672666549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd4013f520 msgr2=0x7fdd400a97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.298+0000 7fdd46519640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd4013f520 0x7fdd400a97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.298+0000 7fdd46519640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.298+0000 7fdd46519640 1 monclient: found mon.noname-a 2024-01-21T04:33:52.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.298+0000 7fdd46519640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.298+0000 7fdd46519640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.298+0000 7fdd46519640 1 -- 172.21.15.69:0/3672666549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c0094d0 con 0x7fdd400720e0 2024-01-21T04:33:52.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.300+0000 7fdd46519640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd400720e0 0x7fdd40141690 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00e930 tx=0x7fdd3c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.300+0000 7fdd36ffd640 10 client.?.objecter ms_handle_connect 0x7fdd400720e0 2024-01-21T04:33:52.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.300+0000 7fdd36ffd640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:52.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 1 -- 172.21.15.69:0/3672666549 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c004030 con 0x7fdd400720e0 2024-01-21T04:33:52.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:52.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 10 monclient: dump: 2024-01-21T04:33:52.305 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 1 -- 172.21.15.69:0/3672666549 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3c004190 con 0x7fdd400720e0 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd4877d640 5 monclient: authenticate success, global_id 4350 2024-01-21T04:33:52.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd4877d640 10 monclient: _renew_subs 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd4877d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd4877d640 1 -- 172.21.15.69:0/3672666549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd38009a70 con 0x7fdd400720e0 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 1 -- 172.21.15.69:0/3672666549 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c00c980 con 0x7fdd400720e0 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd36ffd640 10 monclient: dump: 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.307 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.308 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd44d16640 4 set_mon_vals no callback set 2024-01-21T04:33:52.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd4877d640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:52.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd4877d640 10 monclient: _renew_subs 2024-01-21T04:33:52.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd4877d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd4877d640 1 -- 172.21.15.69:0/3672666549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd400a9ee0 con 0x7fdd400720e0 2024-01-21T04:33:52.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.304+0000 7fdd4877d640 1 librados: init done 2024-01-21T04:33:52.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.305+0000 7fdd4877d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.305+0000 7fdd4877d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.305+0000 7fdd4877d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.305+0000 7fdd4877d640 1 -- 172.21.15.69:0/3672666549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4004e970 con 0x7fdd400720e0 2024-01-21T04:33:52.313 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-21T04:33:52.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.324+0000 7fdd36ffd640 1 -- 172.21.15.69:0/3672666549 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fdd3c00cae0 con 0x7fdd400720e0 2024-01-21T04:33:52.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.325+0000 7fdd36ffd640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdd2406c8b0 0x7fdd2406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.325+0000 7fdd45d18640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdd2406c8b0 0x7fdd2406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.325+0000 7fdd45d18640 10 monclient: get_auth_request con 0x7fdd2406c8b0 auth_method 0 2024-01-21T04:33:52.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.325+0000 7fdd45d18640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdd2406c8b0 0x7fdd2406ed70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdd400acdb0 tx=0x7fdd300091c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.325+0000 7fdd36ffd640 10 client.4350.objecter ms_handle_connect 0x7fdd2406c8b0 2024-01-21T04:33:52.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.328+0000 7fdd36ffd640 1 -- 172.21.15.69:0/3672666549 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdd3c0589a0 con 0x7fdd400720e0 2024-01-21T04:33:52.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.328+0000 7fdd36ffd640 10 client.4350.objecter ms_dispatch 0x7fdd40000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:52.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.328+0000 7fdd36ffd640 3 client.4350.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:52.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.328+0000 7fdd36ffd640 3 client.4350.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:52.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.328+0000 7fdd36ffd640 20 client.4350.objecter dump_active .. 0 homeless 2024-01-21T04:33:52.381 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-21T04:33:52.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.389+0000 7f8bfb7fe640 1 -- 172.21.15.69:0/3279896228 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f8bfc02cd50 con 0x7f8c04144f20 2024-01-21T04:33:52.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.389+0000 7f8bfb7fe640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8be406c8b0 0x7f8be406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.390+0000 7f8c02ffd640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8be406c8b0 0x7f8be406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.390+0000 7f8c02ffd640 10 monclient: get_auth_request con 0x7f8be406c8b0 auth_method 0 2024-01-21T04:33:52.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.390+0000 7f8bfb7fe640 10 client.4369.objecter ms_handle_connect 0x7f8be406c8b0 2024-01-21T04:33:52.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.390+0000 7f8c02ffd640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8be406c8b0 0x7f8be406ed70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009aa0 tx=0x7f8bf4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.392+0000 7f8bfb7fe640 1 -- 172.21.15.69:0/3279896228 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8bfc07ae40 con 0x7f8c04144f20 2024-01-21T04:33:52.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.392+0000 7f8bfb7fe640 10 client.4369.objecter ms_dispatch 0x7f8c04000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:52.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.392+0000 7f8bfb7fe640 3 client.4369.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:52.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.392+0000 7f8bfb7fe640 3 client.4369.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:52.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.392+0000 7f8bfb7fe640 20 client.4369.objecter dump_active .. 0 homeless 2024-01-21T04:33:52.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.402+0000 7f8bfb7fe640 1 -- 172.21.15.69:0/3279896228 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8bfc077180 con 0x7f8c04144f20 2024-01-21T04:33:52.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.403+0000 7f8bfb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.403+0000 7f8bfb7fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:52.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.417+0000 7fdc9e1b6640 10 monclient: get_monmap_and_config 2024-01-21T04:33:52.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.417+0000 7fdc9e1b6640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.417+0000 7fdc9e1b6640 10 monclient: monmap: 2024-01-21T04:33:52.420 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.420 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:52.418743+0000 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:52.418743+0000 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.418+0000 7fdc9e1b6640 1 Processor -- start 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.418+0000 7fdc9e1b6640 1 -- start start 2024-01-21T04:33:52.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.418+0000 7fdc9e1b6640 10 monclient: init 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdc980a8b20 0x7fdc98145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc980a8b20 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc980a9d10 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc980b2310 0x7fdc9813f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc980b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc981456f0 con 0x7fdc980a9d10 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc980a98c0 con 0x7fdc980a8b20 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc981460c0 con 0x7fdc980b2310 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc9e1b6640 10 monclient(hunting): authenticate will time out at 1390.760517s 2024-01-21T04:33:52.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.419+0000 7fdc977fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdc980a8b20 0x7fdc98145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc977fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdc980a8b20 0x7fdc98145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59932/0 (socket says 172.21.15.69:59932) 2024-01-21T04:33:52.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc977fe640 1 -- 172.21.15.69:0/2105330700 learned_addr learned my addr 172.21.15.69:0/2105330700 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc977fe640 10 monclient(hunting): get_auth_request con 0x7fdc980a8b20 auth_method 0 2024-01-21T04:33:52.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc977fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc977fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc97fff640 1 --2- 172.21.15.69:0/2105330700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc980b2310 0x7fdc9813f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc97fff640 10 monclient(hunting): get_auth_request con 0x7fdc980b2310 auth_method 0 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc97fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc97fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.420+0000 7fdc97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.422+0000 7fdc977fe640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2024-01-21T04:33:52.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.422+0000 7fdc977fe640 1 -- 172.21.15.69:0/2105330700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc980b2310 msgr2=0x7fdc9813f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.422+0000 7fdc977fe640 1 --2- 172.21.15.69:0/2105330700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc980b2310 0x7fdc9813f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.422+0000 7fdc977fe640 1 -- 172.21.15.69:0/2105330700 >> v1:172.21.15.69:6789/0 conn(0x7fdc980a9d10 legacy=0x7fdc9813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.422+0000 7fdc977fe640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.422+0000 7fdc977fe640 1 monclient: found mon.noname-c 2024-01-21T04:33:52.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.422+0000 7fdc977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.422+0000 7fdc977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.422+0000 7fdc977fe640 1 -- 172.21.15.69:0/2105330700 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc98146230 con 0x7fdc980a8b20 2024-01-21T04:33:52.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.424+0000 7fdc977fe640 1 --2- 172.21.15.69:0/2105330700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc980a8b20 0x7fdc98145b80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009870 tx=0x7fdc8802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da6519985d55e017 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.427+0000 7fdc967fc640 1 -- 172.21.15.69:0/2105330700 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc880317c0 con 0x7fdc980a8b20 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 10 monclient: dump: 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.429 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 1 -- 172.21.15.69:0/2105330700 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc88031920 con 0x7fdc980a8b20 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc9e1b6640 5 monclient: authenticate success, global_id 4372 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc9e1b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc9e1b6640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 1 -- 172.21.15.69:0/2105330700 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88031cb0 con 0x7fdc980a8b20 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc9e1b6640 4 set_mon_vals no callback set 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:52.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc967fc640 10 monclient: dump: 2024-01-21T04:33:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.431 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.431 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc9e1b6640 10 monclient: shutdown 2024-01-21T04:33:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc9e1b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2105330700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc980a8b20 msgr2=0x7fdc98145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.428+0000 7fdc9e1b6640 1 --2- 172.21.15.69:0/2105330700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc980a8b20 0x7fdc98145b80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009870 tx=0x7fdc8802b860 comp rx=0 tx=0).stop 2024-01-21T04:33:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.429+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2105330700 shutdown_connections 2024-01-21T04:33:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.429+0000 7fdc9e1b6640 1 --2- 172.21.15.69:0/2105330700 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc980b2310 0x7fdc9813f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.429+0000 7fdc9e1b6640 1 --2- 172.21.15.69:0/2105330700 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc980a8b20 0x7fdc98145b80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.429+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2105330700 >> 172.21.15.69:0/2105330700 conn(0x7fdc98066fb0 msgr2=0x7fdc980b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.429+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2105330700 shutdown_connections 2024-01-21T04:33:52.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.429+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2105330700 wait complete. 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.429+0000 7fdc9e1b6640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.429+0000 7fdc9e1b6640 10 monclient: monmap: 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:52.436 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.430+0000 7fdc9e1b6640 1 librados: starting msgr at 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.430+0000 7fdc9e1b6640 1 librados: starting objecter 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.430+0000 7fdc9e1b6640 1 Processor -- start 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.430+0000 7fdc9e1b6640 1 -- start start 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.430+0000 7fdc9e1b6640 1 librados: setting wanted keys 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.430+0000 7fdc9e1b6640 1 librados: calling monclient init 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.430+0000 7fdc9e1b6640 10 monclient: init 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.430+0000 7fdc9e1b6640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc980a8b20 0x7fdc98140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc980a8b20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc980a9d10 0x7fdc981405d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 10 monclient(hunting): picked mon.noname-a con 0x7fdc980a9d10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc980b2310 0x7fdc98132620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc980b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc98140b10 con 0x7fdc980a9d10 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc98141b60 con 0x7fdc980a8b20 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc981426e0 con 0x7fdc980b2310 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc9e1b6640 10 monclient(hunting): authenticate will time out at 1390.772299s 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc977fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc980a8b20 0x7fdc98140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.431+0000 7fdc97fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc980b2310 0x7fdc98132620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc977fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc980a8b20 0x7fdc98140f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:59934/0 (socket says 172.21.15.69:59934) 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc977fe640 1 -- 172.21.15.69:0/2839231803 learned_addr learned my addr 172.21.15.69:0/2839231803 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc977fe640 10 monclient(hunting): get_auth_request con 0x7fdc980a8b20 auth_method 0 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc977fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc977fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc97fff640 10 monclient(hunting): get_auth_request con 0x7fdc980b2310 auth_method 0 2024-01-21T04:33:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc97fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc97fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.432+0000 7fdc97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.435+0000 7fdc977fe640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2024-01-21T04:33:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.435+0000 7fdc977fe640 1 -- 172.21.15.69:0/2839231803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc980b2310 msgr2=0x7fdc98132620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.435+0000 7fdc977fe640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc980b2310 0x7fdc98132620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.435+0000 7fdc977fe640 1 -- 172.21.15.69:0/2839231803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc980a9d10 msgr2=0x7fdc981405d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.435+0000 7fdc977fe640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc980a9d10 0x7fdc981405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.435+0000 7fdc977fe640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.435+0000 7fdc977fe640 1 monclient: found mon.noname-b 2024-01-21T04:33:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.435+0000 7fdc977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.435+0000 7fdc977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.435+0000 7fdc977fe640 1 -- 172.21.15.69:0/2839231803 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc880094d0 con 0x7fdc980a8b20 2024-01-21T04:33:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.439+0000 7fdc977fe640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc980a8b20 0x7fdc98140f40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdc880090b0 tx=0x7fdc880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.439+0000 7fdc7ffff640 10 client.?.objecter ms_handle_connect 0x7fdc980a8b20 2024-01-21T04:33:52.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.439+0000 7fdc7ffff640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.440+0000 7fdc7ffff640 1 -- 172.21.15.69:0/2839231803 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88032860 con 0x7fdc980a8b20 2024-01-21T04:33:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.440+0000 7fdc7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.440+0000 7fdc7ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.440+0000 7fdc7ffff640 10 monclient: dump: 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.440+0000 7fdc7ffff640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.440+0000 7fdc7ffff640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.440+0000 7fdc7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.440+0000 7fdc9e1b6640 5 monclient: authenticate success, global_id 4375 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.440+0000 7fdc9e1b6640 10 monclient: _renew_subs 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.440+0000 7fdc9e1b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.440+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2839231803 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc98142950 con 0x7fdc980a8b20 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.441+0000 7fdc9e1b6640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.441+0000 7fdc9e1b6640 10 monclient: _renew_subs 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.441+0000 7fdc9e1b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.441+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2839231803 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc981430c0 con 0x7fdc980a8b20 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.441+0000 7fdc9e1b6640 1 librados: init done 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.442+0000 7fdc7ffff640 1 -- 172.21.15.69:0/2839231803 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc880329c0 con 0x7fdc980a8b20 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.442+0000 7fdc7ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.442+0000 7fdc7ffff640 1 -- 172.21.15.69:0/2839231803 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88032d30 con 0x7fdc980a8b20 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.442+0000 7fdc7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.442+0000 7fdc7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.442+0000 7fdc7ffff640 10 monclient: dump: 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.442+0000 7fdc95ffb640 4 set_mon_vals no callback set 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.442+0000 7fdc9e1b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.442+0000 7fdc9e1b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.442+0000 7fdc9e1b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.442+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2839231803 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc78005080 con 0x7fdc980a8b20 2024-01-21T04:33:52.451 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-21T04:33:52.485 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-21T04:33:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.491+0000 7fc23e705640 10 monclient: get_monmap_and_config 2024-01-21T04:33:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.491+0000 7fc23e705640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.491+0000 7fc23e705640 10 monclient: monmap: 2024-01-21T04:33:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:52.492105+0000 2024-01-21T04:33:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:52.492105+0000 2024-01-21T04:33:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:52.492 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:52.493 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:52.493 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:52.493 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:52.493 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.491+0000 7fc23e705640 1 Processor -- start 2024-01-21T04:33:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.491+0000 7fc23e705640 1 -- start start 2024-01-21T04:33:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.491+0000 7fc23e705640 10 monclient: init 2024-01-21T04:33:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-01-21T04:33:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2380a8b20 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2380a9d10 0x7fc23813f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 10 monclient(hunting): picked mon.noname-b con 0x7fc2380a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2380b2310 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:52.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2381456f0 con 0x7fc2380a8b20 2024-01-21T04:33:52.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2380a98c0 con 0x7fc2380b2310 2024-01-21T04:33:52.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc238146290 con 0x7fc2380a9d10 2024-01-21T04:33:52.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.492+0000 7fc23e705640 10 monclient(hunting): authenticate will time out at 1390.833784s 2024-01-21T04:33:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.493+0000 7fc2377fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2380a9d10 0x7fc23813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.493+0000 7fc2377fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2380a9d10 0x7fc23813f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:47018/0 (socket says 172.21.15.69:47018) 2024-01-21T04:33:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.493+0000 7fc2377fe640 1 -- 172.21.15.69:0/2134823971 learned_addr learned my addr 172.21.15.69:0/2134823971 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.493+0000 7fc2377fe640 10 monclient(hunting): get_auth_request con 0x7fc2380a9d10 auth_method 0 2024-01-21T04:33:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.493+0000 7fc2377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.493+0000 7fc2377fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.493+0000 7fc2377fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.494+0000 7fc2377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.494+0000 7fc2377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.494+0000 7fc2377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.497+0000 7fc2377fe640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-01-21T04:33:52.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.497+0000 7fc2377fe640 1 -- 172.21.15.69:0/2134823971 >> v1:172.21.15.144:6789/0 conn(0x7fc2380b2310 legacy=0x7fc23813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.497+0000 7fc2377fe640 1 -- 172.21.15.69:0/2134823971 >> v1:172.21.15.69:6789/0 conn(0x7fc2380a8b20 legacy=0x7fc238145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.497+0000 7fc2377fe640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.497+0000 7fc2377fe640 1 monclient: found mon.noname-b 2024-01-21T04:33:52.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.497+0000 7fc2377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.497+0000 7fc2377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:33:52.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.497+0000 7fc2377fe640 1 -- 172.21.15.69:0/2134823971 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2381463d0 con 0x7fc2380a9d10 2024-01-21T04:33:52.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.499+0000 7fc2377fe640 1 --2- 172.21.15.69:0/2134823971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2380a9d10 0x7fc23813f080 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0098a0 tx=0x7fc22c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=339368eed74f5a4a server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 1 -- 172.21.15.69:0/2134823971 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0317c0 con 0x7fc2380a9d10 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 10 monclient: dump: 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.506 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 1 -- 172.21.15.69:0/2134823971 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c031920 con 0x7fc2380a9d10 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 1 -- 172.21.15.69:0/2134823971 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c031cb0 con 0x7fc2380a9d10 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc23e705640 5 monclient: authenticate success, global_id 4313 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc23e705640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc23e705640 4 set_mon_vals no callback set 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.505+0000 7fc236ffd640 10 monclient: dump: 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.507 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.508 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.508 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.508 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.508 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.508 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.508 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.508 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.506+0000 7fc23e705640 10 monclient: shutdown 2024-01-21T04:33:52.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.506+0000 7fc23e705640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:52.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.506+0000 7fc23e705640 1 -- 172.21.15.69:0/2134823971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2380a9d10 msgr2=0x7fc23813f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.506+0000 7fc23e705640 1 --2- 172.21.15.69:0/2134823971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2380a9d10 0x7fc23813f080 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0098a0 tx=0x7fc22c02b860 comp rx=0 tx=0).stop 2024-01-21T04:33:52.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.506+0000 7fc23e705640 1 -- 172.21.15.69:0/2134823971 shutdown_connections 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.506+0000 7fc23e705640 1 --2- 172.21.15.69:0/2134823971 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc2380a9d10 0x7fc23813f080 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.506+0000 7fc23e705640 1 -- 172.21.15.69:0/2134823971 >> 172.21.15.69:0/2134823971 conn(0x7fc238066fb0 msgr2=0x7fc2380b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.507+0000 7fc23e705640 1 -- 172.21.15.69:0/2134823971 shutdown_connections 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.507+0000 7fc23e705640 1 -- 172.21.15.69:0/2134823971 wait complete. 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.508+0000 7fc23e705640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.508+0000 7fc23e705640 10 monclient: monmap: 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:52.510 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:52.511 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.508+0000 7fc23e705640 1 librados: starting msgr at 2024-01-21T04:33:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.508+0000 7fc23e705640 1 librados: starting objecter 2024-01-21T04:33:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.509+0000 7fc23e705640 1 Processor -- start 2024-01-21T04:33:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.509+0000 7fc23e705640 1 -- start start 2024-01-21T04:33:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.509+0000 7fc23e705640 1 librados: setting wanted keys 2024-01-21T04:33:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.509+0000 7fc23e705640 1 librados: calling monclient init 2024-01-21T04:33:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.509+0000 7fc23e705640 10 monclient: init 2024-01-21T04:33:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:33:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2380a8b20 0x7fc238137130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 10 monclient(hunting): picked mon.noname-a con 0x7fc2380a8b20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2380b2310 0x7fc238140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 10 monclient(hunting): picked mon.noname-b con 0x7fc2380b2310 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc238137a40 0x7fc238140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 10 monclient(hunting): picked mon.noname-c con 0x7fc238137a40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2381404d0 con 0x7fc2380a8b20 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc238138ca0 con 0x7fc2380b2310 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc238137670 con 0x7fc238137a40 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.510+0000 7fc23e705640 10 monclient(hunting): authenticate will time out at 1390.852145s 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.511+0000 7fc23cca2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc238137a40 0x7fc238140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.512+0000 7fc23cca2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc238137a40 0x7fc238140ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:47032/0 (socket says 172.21.15.69:47032) 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.512+0000 7fc23cca2640 1 -- 172.21.15.69:0/365757925 learned_addr learned my addr 172.21.15.69:0/365757925 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.512+0000 7fc23cca2640 10 monclient(hunting): get_auth_request con 0x7fc238137a40 auth_method 0 2024-01-21T04:33:52.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.512+0000 7fc23cca2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.512+0000 7fc23cca2640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.512+0000 7fc23cca2640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.512+0000 7fc23cca2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.512+0000 7fc23cca2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.512+0000 7fc23cca2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.517+0000 7fc23cca2640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-01-21T04:33:52.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.517+0000 7fc23cca2640 1 -- 172.21.15.69:0/365757925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2380b2310 msgr2=0x7fc238140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.517+0000 7fc23cca2640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2380b2310 0x7fc238140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.517+0000 7fc23cca2640 1 -- 172.21.15.69:0/365757925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2380a8b20 msgr2=0x7fc238137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.517+0000 7fc23cca2640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2380a8b20 0x7fc238137130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.517+0000 7fc23cca2640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.517+0000 7fc23cca2640 1 monclient: found mon.noname-c 2024-01-21T04:33:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.517+0000 7fc23cca2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.517+0000 7fc23cca2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T04:33:52.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.517+0000 7fc23cca2640 1 -- 172.21.15.69:0/365757925 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c0094d0 con 0x7fc238137a40 2024-01-21T04:33:52.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.521+0000 7fc23cca2640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc238137a40 0x7fc238140ea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc23000b9e0 tx=0x7fc23000bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.521+0000 7fc234ff9640 10 client.?.objecter ms_handle_connect 0x7fc238137a40 2024-01-21T04:33:52.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.521+0000 7fc234ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:52.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 1 -- 172.21.15.69:0/365757925 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23000cb20 con 0x7fc238137a40 2024-01-21T04:33:52.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:52.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 10 monclient: dump: 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 1 -- 172.21.15.69:0/365757925 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23000cc80 con 0x7fc238137a40 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 1 -- 172.21.15.69:0/365757925 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc230014680 con 0x7fc238137a40 2024-01-21T04:33:52.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc234ff9640 10 monclient: dump: 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc2367fc640 4 set_mon_vals no callback set 2024-01-21T04:33:52.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.527+0000 7fc23e705640 5 monclient: authenticate success, global_id 4316 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.528+0000 7fc23e705640 10 monclient: _renew_subs 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.528+0000 7fc23e705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.528+0000 7fc23e705640 1 -- 172.21.15.69:0/365757925 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2381413e0 con 0x7fc238137a40 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.528+0000 7fc23e705640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.528+0000 7fc23e705640 10 monclient: _renew_subs 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.528+0000 7fc23e705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.528+0000 7fc23e705640 1 -- 172.21.15.69:0/365757925 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2381db050 con 0x7fc238137a40 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.528+0000 7fc23e705640 1 librados: init done 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.529+0000 7fc23e705640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.529+0000 7fc23e705640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.529+0000 7fc23e705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:52.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.529+0000 7fc23e705640 1 -- 172.21.15.69:0/365757925 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2381402a0 con 0x7fc238137a40 2024-01-21T04:33:52.557 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-21T04:33:52.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.586+0000 7f423415e640 10 monclient: get_monmap_and_config 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.586+0000 7f423415e640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.586+0000 7f423415e640 10 monclient: monmap: 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:52.587865+0000 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:52.587865+0000 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:52.588 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:52.589 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:52.589 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.587+0000 7f423415e640 1 Processor -- start 2024-01-21T04:33:52.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.588+0000 7f423415e640 1 -- start start 2024-01-21T04:33:52.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.588+0000 7f423415e640 10 monclient: init 2024-01-21T04:33:52.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.588+0000 7f423415e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.588+0000 7f423415e640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.588+0000 7f423415e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f422c0a8b20 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.589+0000 7f423415e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f422c0a9d10 0x7f422c13f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.589+0000 7f423415e640 10 monclient(hunting): picked mon.noname-c con 0x7f422c0a9d10 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.589+0000 7f423415e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f422c0b2310 0x7f422c13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.589+0000 7f423415e640 10 monclient(hunting): picked mon.noname-b con 0x7f422c0b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.589+0000 7f423415e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422c1456f0 con 0x7f422c0a8b20 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.589+0000 7f423415e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.589+0000 7f423415e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422c0a98c0 con 0x7f422c0a9d10 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.589+0000 7f423415e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.589+0000 7f423415e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c13fb00 con 0x7f422c0b2310 2024-01-21T04:33:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.589+0000 7f423415e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.589+0000 7f423415e640 10 monclient(hunting): authenticate will time out at 1390.930438s 2024-01-21T04:33:52.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.600+0000 7fdd36ffd640 1 -- 172.21.15.69:0/3672666549 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fdd3c054ce0 con 0x7fdd400720e0 2024-01-21T04:33:52.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.600+0000 7fdd36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.600+0000 7fdd36ffd640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:52.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.609+0000 7f6b4e0a2640 10 monclient: get_monmap_and_config 2024-01-21T04:33:52.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.609+0000 7f6b4e0a2640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.609+0000 7f6b4e0a2640 10 monclient: monmap: 2024-01-21T04:33:52.610 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.610 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.610 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:52.610367+0000 2024-01-21T04:33:52.610 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:52.610367+0000 2024-01-21T04:33:52.610 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.610 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.610 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:52.610 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:52.611 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:52.611 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:52.611 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:52.611 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.610+0000 7f6b4e0a2640 1 Processor -- start 2024-01-21T04:33:52.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.610+0000 7f6b4e0a2640 1 -- start start 2024-01-21T04:33:52.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.610+0000 7f6b4e0a2640 10 monclient: init 2024-01-21T04:33:52.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.611+0000 7f6b4e0a2640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.611+0000 7f6b4e0a2640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-01-21T04:33:52.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.611+0000 7f6b4e0a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b480a8b20 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:52.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.612+0000 7f6b4e0a2640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6b480a9d10 0x7f6b4813f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.612+0000 7f6b4e0a2640 10 monclient(hunting): picked mon.noname-a con 0x7f6b480a9d10 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:52.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.612+0000 7f6b4e0a2640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6b480b2310 0x7f6b4813f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.612+0000 7f6b4e0a2640 10 monclient(hunting): picked mon.noname-c con 0x7f6b480b2310 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:52.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.612+0000 7f6b4e0a2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b481456f0 con 0x7f6b480a8b20 2024-01-21T04:33:52.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.612+0000 7f6b4e0a2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.612+0000 7f6b4e0a2640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b480a98c0 con 0x7f6b480a9d10 2024-01-21T04:33:52.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.612+0000 7f6b4e0a2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.612+0000 7f6b4e0a2640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4813fb00 con 0x7f6b480b2310 2024-01-21T04:33:52.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.612+0000 7f6b4e0a2640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.612+0000 7f6b4e0a2640 10 monclient(hunting): authenticate will time out at 1390.954016s 2024-01-21T04:33:52.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.618+0000 7f4231efa640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f422c0a8b20 0x7f422c145b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58988/0 (socket says 172.21.15.69:58988) 2024-01-21T04:33:52.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.618+0000 7f4231efa640 1 -- 172.21.15.69:0/835889553 learned_addr learned my addr 172.21.15.69:0/835889553 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.621+0000 7f6b46ffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6b480a9d10 0x7f6b4813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.623+0000 7f6b46ffd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6b480a9d10 0x7f6b4813f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53540/0 (socket says 172.21.15.69:53540) 2024-01-21T04:33:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.623+0000 7f6b46ffd640 1 -- 172.21.15.69:0/2582096578 learned_addr learned my addr 172.21.15.69:0/2582096578 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.623+0000 7f6b46ffd640 10 monclient(hunting): get_auth_request con 0x7f6b480a9d10 auth_method 0 2024-01-21T04:33:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.623+0000 7f6b46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.623+0000 7f6b46ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.623+0000 7f6b46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.624+0000 7f6b46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.624+0000 7f6b46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.624+0000 7f6b46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.625+0000 7f8c09415640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T04:33:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.625+0000 7f8c09415640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T04:33:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.625+0000 7f8c09415640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.625+0000 7f8c09415640 1 -- 172.21.15.69:0/3279896228 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f8c04145fd0 con 0x7f8c04144f20 2024-01-21T04:33:52.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.630+0000 7f6b46ffd640 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2024-01-21T04:33:52.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.630+0000 7f6b46ffd640 1 -- 172.21.15.69:0/2582096578 >> v2:172.21.15.144:3300/0 conn(0x7f6b480b2310 msgr2=0x7f6b4813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.630+0000 7f6b46ffd640 1 --2- 172.21.15.69:0/2582096578 >> v2:172.21.15.144:3300/0 conn(0x7f6b480b2310 0x7f6b4813f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.630+0000 7f6b46ffd640 1 -- 172.21.15.69:0/2582096578 >> v1:172.21.15.144:6789/0 conn(0x7f6b480a8b20 legacy=0x7f6b48145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.630+0000 7f6b46ffd640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.630+0000 7f6b46ffd640 1 monclient: found mon.noname-a 2024-01-21T04:33:52.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.630+0000 7f6b46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.630+0000 7f6b46ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.631+0000 7f6b46ffd640 1 -- 172.21.15.69:0/2582096578 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b48146290 con 0x7f6b480a9d10 2024-01-21T04:33:52.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.633+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285918556 0 0) 0x7f422c1456f0 con 0x7f422c0a8b20 2024-01-21T04:33:52.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.633+0000 7f4230ef8640 10 monclient(hunting): _negotiate 2024-01-21T04:33:52.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.633+0000 7f4230ef8640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.633+0000 7f4230ef8640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.633+0000 7f4230ef8640 10 monclient(hunting): my global_id is 4356 2024-01-21T04:33:52.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.633+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4218003520 con 0x7f422c0a8b20 2024-01-21T04:33:52.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.634+0000 7f6b46ffd640 1 --2- 172.21.15.69:0/2582096578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b480a9d10 0x7f6b4813f080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b480aa150 tx=0x7f6b3c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77eda42745de822 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.642+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 809656761 0 0) 0x7f4218003520 con 0x7f422c0a8b20 2024-01-21T04:33:52.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f422c0b2310 msgr2=0x7f422c13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 --2- 172.21.15.69:0/835889553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f422c0b2310 0x7f422c13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 >> v2:172.21.15.144:3300/0 conn(0x7f422c0a9d10 msgr2=0x7f422c13f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 --2- 172.21.15.69:0/835889553 >> v2:172.21.15.144:3300/0 conn(0x7f422c0a9d10 0x7f422c13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 monclient: found mon.noname-a-legacy 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.69:6789/0 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c146290 con 0x7f422c0a8b20 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 125097436 0 0) 0x7f4228001670 con 0x7f422c0a8b20 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f423415e640 5 monclient: authenticate success, global_id 4356 2024-01-21T04:33:52.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f423415e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient: dump: 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.648 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] but i'm connected to v1:172.21.15.69:6789/0, reconnecting 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 >> v1:172.21.15.69:6789/0 conn(0x7f422c0a8b20 legacy=0x7f422c145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 --2- 172.21.15.69:0/835889553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f422c0b2310 0x7f4218003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient(hunting): picked mon.c con 0x7f422c0b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 --2- 172.21.15.69:0/835889553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f42180040d0 0x7f42180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient(hunting): picked mon.b con 0x7f42180040d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 --2- 172.21.15.69:0/835889553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 0x7f421800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient(hunting): picked mon.a con 0x7f4218004e20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0a98c0 con 0x7f4218004e20 2024-01-21T04:33:52.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.643+0000 7f4230ef8640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.644+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c13fb00 con 0x7f42180040d0 2024-01-21T04:33:52.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.644+0000 7f4230ef8640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.644+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42180055a0 con 0x7f422c0b2310 2024-01-21T04:33:52.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.644+0000 7f4230ef8640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.644+0000 7f4230ef8640 10 monclient(hunting): _finish_auth 0 2024-01-21T04:33:52.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.644+0000 7f423415e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:52.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 1 -- 172.21.15.69:0/2582096578 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c0317e0 con 0x7f6b480a9d10 2024-01-21T04:33:52.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 10 monclient: dump: 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 1 -- 172.21.15.69:0/2582096578 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c031940 con 0x7f6b480a9d10 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 1 -- 172.21.15.69:0/2582096578 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c031cd0 con 0x7f6b480a9d10 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.645+0000 7f6b467fc640 10 monclient: dump: 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.652 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.646+0000 7f6b4e0a2640 5 monclient: authenticate success, global_id 4353 2024-01-21T04:33:52.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.646+0000 7f6b4e0a2640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:52.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.646+0000 7f6b4e0a2640 4 set_mon_vals no callback set 2024-01-21T04:33:52.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.646+0000 7f42326fb640 1 --2- 172.21.15.69:0/835889553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 0x7f421800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.646+0000 7f6b4e0a2640 10 monclient: shutdown 2024-01-21T04:33:52.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.646+0000 7f6b4e0a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:52.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.646+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2582096578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b480a9d10 msgr2=0x7f6b4813f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.646+0000 7f6b4e0a2640 1 --2- 172.21.15.69:0/2582096578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b480a9d10 0x7f6b4813f080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b480aa150 tx=0x7f6b3c02b880 comp rx=0 tx=0).stop 2024-01-21T04:33:52.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.647+0000 7f42326fb640 10 monclient(hunting): get_auth_request con 0x7f4218004e20 auth_method 0 2024-01-21T04:33:52.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.647+0000 7f42326fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.647+0000 7f42326fb640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.647+0000 7f42326fb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T04:33:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.647+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2582096578 shutdown_connections 2024-01-21T04:33:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.647+0000 7f6b4e0a2640 1 --2- 172.21.15.69:0/2582096578 >> v2:172.21.15.144:3300/0 conn(0x7f6b480b2310 0x7f6b4813f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.647+0000 7f6b4e0a2640 1 --2- 172.21.15.69:0/2582096578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b480a9d10 0x7f6b4813f080 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.647+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2582096578 >> 172.21.15.69:0/2582096578 conn(0x7f6b48066fb0 msgr2=0x7f6b480b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.647+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2582096578 shutdown_connections 2024-01-21T04:33:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.647+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2582096578 wait complete. 2024-01-21T04:33:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f6b4e0a2640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f6b4e0a2640 10 monclient: monmap: 2024-01-21T04:33:52.656 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.656 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:52.656 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:52.656 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:52.656 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.656 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.656 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:52.656 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:52.656 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:52.656 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f42326fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f42326fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f42326fb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T04:33:52.657 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-21T04:33:52.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f6b4e0a2640 1 librados: starting msgr at 2024-01-21T04:33:52.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f6b4e0a2640 1 librados: starting objecter 2024-01-21T04:33:52.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f6b4e0a2640 1 Processor -- start 2024-01-21T04:33:52.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f6b4e0a2640 1 -- start start 2024-01-21T04:33:52.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f6b4e0a2640 1 librados: setting wanted keys 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f6b4e0a2640 1 librados: calling monclient init 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.648+0000 7f6b4e0a2640 10 monclient: init 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b480a8b20 0x7f6b48140bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 10 monclient(hunting): picked mon.noname-c con 0x7f6b480a8b20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b480a9d10 0x7f6b4813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 10 monclient(hunting): picked mon.noname-a con 0x7f6b480a9d10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b480b2310 0x7f6b4813abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 10 monclient(hunting): picked mon.noname-b con 0x7f6b480b2310 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b48143050 con 0x7f6b480a9d10 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b481418b0 con 0x7f6b480b2310 2024-01-21T04:33:52.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4813fd60 con 0x7f6b480a8b20 2024-01-21T04:33:52.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.649+0000 7f6b4e0a2640 10 monclient(hunting): authenticate will time out at 1390.990930s 2024-01-21T04:33:52.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.651+0000 7f42326fb640 10 monclient(hunting): handle_auth_done global_id 4356 payload 293 2024-01-21T04:33:52.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.651+0000 7f42326fb640 1 -- 172.21.15.69:0/835889553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f422c0b2310 msgr2=0x7f4218003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.652+0000 7f42326fb640 1 --2- 172.21.15.69:0/835889553 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f422c0b2310 0x7f4218003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.652+0000 7f42326fb640 1 -- 172.21.15.69:0/835889553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f42180040d0 msgr2=0x7f42180047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.652+0000 7f42326fb640 1 --2- 172.21.15.69:0/835889553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f42180040d0 0x7f42180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.652+0000 7f42326fb640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.652+0000 7f42326fb640 1 monclient: found mon.a 2024-01-21T04:33:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.652+0000 7f42326fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.652+0000 7f42326fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.652+0000 7f42326fb640 1 -- 172.21.15.69:0/835889553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4218003520 con 0x7f4218004e20 2024-01-21T04:33:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.652+0000 7f42326fb640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.652+0000 7f42326fb640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.652+0000 7f42326fb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.653+0000 7f6b46ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b480a9d10 0x7f6b4813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.655+0000 7f6b46ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b480a9d10 0x7f6b4813a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53560/0 (socket says 172.21.15.69:53560) 2024-01-21T04:33:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.655+0000 7f6b46ffd640 1 -- 172.21.15.69:0/2581982995 learned_addr learned my addr 172.21.15.69:0/2581982995 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.655+0000 7f6b46ffd640 10 monclient(hunting): get_auth_request con 0x7f6b480a9d10 auth_method 0 2024-01-21T04:33:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.655+0000 7f6b46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.655+0000 7f6b46ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.655+0000 7f6b46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.655+0000 7f42326fb640 1 --2- 172.21.15.69:0/835889553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 0x7f421800b980 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4224009e40 tx=0x7f42180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec641e11909ca8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.656+0000 7f6b46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.656+0000 7f6b46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.656+0000 7f6b46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.658+0000 7fc234ff9640 1 -- 172.21.15.69:0/365757925 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc23001d3b0 con 0x7fc238137a40 2024-01-21T04:33:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.659+0000 7fc234ff9640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc22406c860 0x7fc22406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.659+0000 7fc237fff640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc22406c860 0x7fc22406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.659+0000 7fc237fff640 10 monclient: get_auth_request con 0x7fc22406c860 auth_method 0 2024-01-21T04:33:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7fc234ff9640 10 client.4316.objecter ms_handle_connect 0x7fc22406c860 2024-01-21T04:33:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7fc237fff640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc22406c860 0x7fc22406ed20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc228001560 tx=0x7fc228007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7f6b46ffd640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-01-21T04:33:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7f6b46ffd640 1 -- 172.21.15.69:0/2581982995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b480a8b20 msgr2=0x7f6b48140bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7f6b46ffd640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b480a8b20 0x7f6b48140bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7f6b46ffd640 1 -- 172.21.15.69:0/2581982995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b480b2310 msgr2=0x7f6b4813abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7f6b46ffd640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b480b2310 0x7f6b4813abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7f6b46ffd640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7f6b46ffd640 1 monclient: found mon.noname-a 2024-01-21T04:33:52.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7f6b46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7f6b46ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.660+0000 7f6b46ffd640 1 -- 172.21.15.69:0/2581982995 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c0094d0 con 0x7f6b480a9d10 2024-01-21T04:33:52.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.665+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422402cd90 con 0x7f4218004e20 2024-01-21T04:33:52.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.665+0000 7f4230ef8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.665+0000 7f4230ef8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:52.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.665+0000 7f4230ef8640 10 monclient: dump: 2024-01-21T04:33:52.667 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.667 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.667 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.667 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.667 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.665+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4224034440 con 0x7f4218004e20 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.665+0000 7f423415e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.665+0000 7f4230ef8640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.665+0000 7f4230ef8640 1 -- 172.21.15.69:0/835889553 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42240347f0 con 0x7f4218004e20 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.665+0000 7f423415e640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.665+0000 7f423415e640 4 set_mon_vals no callback set 2024-01-21T04:33:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.666+0000 7f4230ef8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.666+0000 7f4230ef8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.666+0000 7f4230ef8640 10 monclient: dump: 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.669 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.666+0000 7f423415e640 10 monclient: shutdown 2024-01-21T04:33:52.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.666+0000 7f423415e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:52.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.666+0000 7f423415e640 1 -- 172.21.15.69:0/835889553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 msgr2=0x7f421800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.666+0000 7f423415e640 1 --2- 172.21.15.69:0/835889553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 0x7f421800b980 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4224009e40 tx=0x7f42180057f0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.666+0000 7f42326fb640 1 -- 172.21.15.69:0/835889553 reap_dead start 2024-01-21T04:33:52.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.666+0000 7f423415e640 1 -- 172.21.15.69:0/835889553 shutdown_connections 2024-01-21T04:33:52.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.667+0000 7f423415e640 1 -- 172.21.15.69:0/835889553 >> 172.21.15.69:0/835889553 conn(0x7f422c066fb0 msgr2=0x7f422c0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:52.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.667+0000 7f423415e640 1 -- 172.21.15.69:0/835889553 shutdown_connections 2024-01-21T04:33:52.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.667+0000 7f423415e640 1 -- 172.21.15.69:0/835889553 wait complete. 2024-01-21T04:33:52.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.668+0000 7f6b46ffd640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b480a9d10 0x7f6b4813a6b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0090b0 tx=0x7f6b3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.668+0000 7f6b2ffff640 10 client.?.objecter ms_handle_connect 0x7f6b480a9d10 2024-01-21T04:33:52.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.668+0000 7f6b2ffff640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:52.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.668+0000 7f423415e640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.668+0000 7f423415e640 10 monclient: monmap: 2024-01-21T04:33:52.672 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.672 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:52.672 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:52.672 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:52.672 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.672 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.672 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:52.673 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:52.673 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:52.673 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.669+0000 7f423415e640 1 librados: starting msgr at 2024-01-21T04:33:52.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.669+0000 7f423415e640 1 librados: starting objecter 2024-01-21T04:33:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.671+0000 7f423415e640 1 Processor -- start 2024-01-21T04:33:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.671+0000 7f423415e640 1 -- start start 2024-01-21T04:33:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.671+0000 7f423415e640 1 librados: setting wanted keys 2024-01-21T04:33:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.671+0000 7f423415e640 1 librados: calling monclient init 2024-01-21T04:33:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.671+0000 7f423415e640 10 monclient: init 2024-01-21T04:33:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.672+0000 7f423415e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.672+0000 7f423415e640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:33:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.672+0000 7f423415e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 0x7f422c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.672+0000 7f423415e640 10 monclient(hunting): picked mon.noname-a con 0x7f4218004e20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.672+0000 7f423415e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f422c1cd5c0 0x7f422c1ce320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.673+0000 7f423415e640 10 monclient(hunting): picked mon.noname-b con 0x7f422c1cd5c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.673+0000 7f423415e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f422c1cb2b0 0x7f422c1da690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.673+0000 7f423415e640 10 monclient(hunting): picked mon.noname-c con 0x7f422c1cb2b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.673+0000 7f423415e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.673+0000 7f423415e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c0ab780 con 0x7f4218004e20 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.673+0000 7f423415e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.673+0000 7f423415e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c13a7c0 con 0x7f422c1cd5c0 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.673+0000 7f423415e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.673+0000 7f423415e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422c1ce860 con 0x7f422c1cb2b0 2024-01-21T04:33:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.673+0000 7f423415e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.673+0000 7f423415e640 10 monclient(hunting): authenticate will time out at 1391.015103s 2024-01-21T04:33:52.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.675+0000 7f4231efa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 0x7f422c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 1 -- 172.21.15.69:0/2581982995 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c032980 con 0x7f6b480a9d10 2024-01-21T04:33:52.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:52.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 10 monclient: dump: 2024-01-21T04:33:52.678 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.678 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 1 -- 172.21.15.69:0/2581982995 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c032b00 con 0x7f6b480a9d10 2024-01-21T04:33:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b4e0a2640 5 monclient: authenticate success, global_id 4359 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 1 -- 172.21.15.69:0/2581982995 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c0453f0 con 0x7f6b480a9d10 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b2ffff640 10 monclient: dump: 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b4e0a2640 10 monclient: _renew_subs 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b4e0a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2581982995 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b48141130 con 0x7f6b480a9d10 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b45ffb640 4 set_mon_vals no callback set 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b4e0a2640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b4e0a2640 10 monclient: _renew_subs 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b4e0a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2581982995 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b48141600 con 0x7f6b480a9d10 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.676+0000 7f6b4e0a2640 1 librados: init done 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.677+0000 7f4231efa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 0x7f422c0b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53572/0 (socket says 172.21.15.69:53572) 2024-01-21T04:33:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.677+0000 7f4231efa640 1 -- 172.21.15.69:0/935342505 learned_addr learned my addr 172.21.15.69:0/935342505 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.677+0000 7f4231efa640 10 monclient(hunting): get_auth_request con 0x7f4218004e20 auth_method 0 2024-01-21T04:33:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.677+0000 7f4231efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.677+0000 7f4231efa640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.677+0000 7f4231efa640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.677+0000 7f6b4e0a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.677+0000 7f6b4e0a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.678+0000 7f6b4e0a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.678+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2581982995 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4804e970 con 0x7f6b480a9d10 2024-01-21T04:33:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.678+0000 7f4231efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.678+0000 7f4231efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.678+0000 7f4231efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.678+0000 7fc234ff9640 1 -- 172.21.15.69:0/365757925 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc23000ce30 con 0x7fc238137a40 2024-01-21T04:33:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.678+0000 7fc234ff9640 10 client.4316.objecter ms_dispatch 0x7fc238000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.678+0000 7fc234ff9640 3 client.4316.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.678+0000 7fc234ff9640 3 client.4316.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.678+0000 7fc234ff9640 20 client.4316.objecter dump_active .. 0 homeless 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.689+0000 7f941c59e640 10 monclient: get_monmap_and_config 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.689+0000 7f941c59e640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.689+0000 7f941c59e640 10 monclient: monmap: 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:52.690179+0000 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:52.690179+0000 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:52.691 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.689+0000 7f941c59e640 1 Processor -- start 2024-01-21T04:33:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.689+0000 7f941c59e640 1 -- start start 2024-01-21T04:33:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.689+0000 7f941c59e640 10 monclient: init 2024-01-21T04:33:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-01-21T04:33:52.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94140a8b20 0x7f9414145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 10 monclient(hunting): picked mon.noname-b con 0x7f94140a8b20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94140a9d10 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f94140b2310 0x7f941413f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 10 monclient(hunting): picked mon.noname-c con 0x7f94140b2310 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94141456f0 con 0x7f94140a9d10 2024-01-21T04:33:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94140a98c0 con 0x7f94140b2310 2024-01-21T04:33:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94141460c0 con 0x7f94140a8b20 2024-01-21T04:33:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.691+0000 7f941c59e640 10 monclient(hunting): authenticate will time out at 1391.032408s 2024-01-21T04:33:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.693+0000 7f4231efa640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-01-21T04:33:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.693+0000 7f4231efa640 1 -- 172.21.15.69:0/935342505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f422c1cb2b0 msgr2=0x7f422c1da690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.693+0000 7f4231efa640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f422c1cb2b0 0x7f422c1da690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.693+0000 7f4231efa640 1 -- 172.21.15.69:0/935342505 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f422c1cd5c0 msgr2=0x7f422c1ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.693+0000 7f4231efa640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f422c1cd5c0 0x7f422c1ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.693+0000 7f4231efa640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.693+0000 7f4231efa640 1 monclient: found mon.noname-a 2024-01-21T04:33:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.693+0000 7f4231efa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.693+0000 7f4231efa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.693+0000 7f4231efa640 1 -- 172.21.15.69:0/935342505 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c1cde90 con 0x7f4218004e20 2024-01-21T04:33:52.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.696+0000 7fdc7ffff640 1 -- 172.21.15.69:0/2839231803 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fdc8804e450 con 0x7fdc980a8b20 2024-01-21T04:33:52.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.697+0000 7fdc7ffff640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdc8406c890 0x7fdc8406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.697+0000 7fdc96ffd640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdc8406c890 0x7fdc8406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.697+0000 7fdc96ffd640 10 monclient: get_auth_request con 0x7fdc8406c890 auth_method 0 2024-01-21T04:33:52.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.697+0000 7fdc7ffff640 10 client.4375.objecter ms_handle_connect 0x7fdc8406c890 2024-01-21T04:33:52.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.697+0000 7fdc96ffd640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdc8406c890 0x7fdc8406ed50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdc981422c0 tx=0x7fdc8c007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.704+0000 7fdc7ffff640 1 -- 172.21.15.69:0/2839231803 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fdc8807add0 con 0x7fdc980a8b20 2024-01-21T04:33:52.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.704+0000 7fdc7ffff640 10 client.4375.objecter ms_dispatch 0x7fdc98000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:52.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.704+0000 7fdc7ffff640 3 client.4375.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:52.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.704+0000 7fdc7ffff640 3 client.4375.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:52.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.704+0000 7fdc7ffff640 20 client.4375.objecter dump_active .. 0 homeless 2024-01-21T04:33:52.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.710+0000 7f6b2ffff640 1 -- 172.21.15.69:0/2581982995 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6b3c04d3f0 con 0x7f6b480a9d10 2024-01-21T04:33:52.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.710+0000 7f6b2ffff640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b3406c860 0x7f6b3406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.711+0000 7f6b477fe640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b3406c860 0x7f6b3406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.711+0000 7f6b477fe640 10 monclient: get_auth_request con 0x7f6b3406c860 auth_method 0 2024-01-21T04:33:52.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.711+0000 7f6b477fe640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b3406c860 0x7f6b3406ed20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009650 tx=0x7f6b38006ba0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.711+0000 7f6b2ffff640 10 client.4359.objecter ms_handle_connect 0x7f6b3406c860 2024-01-21T04:33:52.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.716+0000 7fdc7ffff640 1 -- 172.21.15.69:0/2839231803 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fdc88076a10 con 0x7fdc980a8b20 2024-01-21T04:33:52.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.716+0000 7fdc7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.716+0000 7fdc7ffff640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:52.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.717+0000 7f6b2ffff640 1 -- 172.21.15.69:0/2581982995 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f6b3c07a840 con 0x7f6b480a9d10 2024-01-21T04:33:52.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.717+0000 7f6b2ffff640 10 client.4359.objecter ms_dispatch 0x7f6b48000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:52.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.717+0000 7f6b2ffff640 3 client.4359.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:52.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.717+0000 7f6b2ffff640 3 client.4359.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:52.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.717+0000 7f6b2ffff640 20 client.4359.objecter dump_active .. 0 homeless 2024-01-21T04:33:52.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.728+0000 7f941ab3b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f94140b2310 0x7f941413f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.730+0000 7f941ab3b640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f94140b2310 0x7f941413f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:60006/0 (socket says 172.21.15.69:60006) 2024-01-21T04:33:52.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.730+0000 7f941ab3b640 1 -- 172.21.15.69:0/3663210260 learned_addr learned my addr 172.21.15.69:0/3663210260 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.730+0000 7f941ab3b640 10 monclient(hunting): get_auth_request con 0x7f94140b2310 auth_method 0 2024-01-21T04:33:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.730+0000 7f941ab3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.730+0000 7f941ab3b640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.730+0000 7f941ab3b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.731+0000 7f8bfb7fe640 1 -- 172.21.15.69:0/3279896228 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8bfc043020 con 0x7f8c04144f20 2024-01-21T04:33:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.731+0000 7f8bfb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-01-21T04:33:52.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.731+0000 7f8bfb7fe640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:52.732 INFO:teuthology.orchestra.run.smithi069.stdout:85899345928 2024-01-21T04:33:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.731+0000 7f941ab3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.731+0000 7f941ab3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.731+0000 7f941ab3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.733+0000 7f8c09415640 10 librados: watch_flush enter 2024-01-21T04:33:52.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.733+0000 7f8c09415640 10 librados: watch_flush exit 2024-01-21T04:33:52.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.733+0000 7f8c09415640 20 client.4369.objecter shutdown clearing up homeless session... 2024-01-21T04:33:52.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.733+0000 7f8c09415640 10 client.4369.objecter successfully canceled tick 2024-01-21T04:33:52.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 -- 172.21.15.69:0/3279896228 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8be406c8b0 msgr2=0x7f8be406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8be406c8b0 0x7f8be406ed70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009aa0 tx=0x7f8bf4009190 comp rx=0 tx=0).stop 2024-01-21T04:33:52.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 10 monclient: shutdown 2024-01-21T04:33:52.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:52.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 -- 172.21.15.69:0/3279896228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c04144f20 msgr2=0x7f8c041d7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c04144f20 0x7f8c041d7760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc032000 tx=0x7f8bfc004a40 comp rx=0 tx=0).stop 2024-01-21T04:33:52.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 -- 172.21.15.69:0/3279896228 shutdown_connections 2024-01-21T04:33:52.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f8be406c8b0 0x7f8be406ed70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8c04144f20 0x7f8c041d7760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c0413f950 0x7f8c041449e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 --2- 172.21.15.69:0/3279896228 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f8c040a8ab0 0x7f8c04143e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 -- 172.21.15.69:0/3279896228 >> 172.21.15.69:0/3279896228 conn(0x7f8c04066f00 msgr2=0x7f8c040b5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:52.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 -- 172.21.15.69:0/3279896228 shutdown_connections 2024-01-21T04:33:52.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 -- 172.21.15.69:0/3279896228 wait complete. 2024-01-21T04:33:52.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.734+0000 7f8c09415640 1 librados: shutdown 2024-01-21T04:33:52.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.735+0000 7f941ab3b640 10 monclient(hunting): handle_auth_done global_id 4378 payload 274 2024-01-21T04:33:52.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.735+0000 7f941ab3b640 1 -- 172.21.15.69:0/3663210260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94140a8b20 msgr2=0x7f9414145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:52.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.735+0000 7f941ab3b640 1 --2- 172.21.15.69:0/3663210260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94140a8b20 0x7f9414145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.735+0000 7f941ab3b640 1 -- 172.21.15.69:0/3663210260 >> v1:172.21.15.144:6789/0 conn(0x7f94140a9d10 legacy=0x7f941413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.735+0000 7f941ab3b640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.735+0000 7f941ab3b640 1 monclient: found mon.noname-c 2024-01-21T04:33:52.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.735+0000 7f941ab3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.735+0000 7f941ab3b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.735+0000 7f941ab3b640 1 -- 172.21.15.69:0/3663210260 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9414146230 con 0x7f94140b2310 2024-01-21T04:33:52.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.739+0000 7f941ab3b640 1 --2- 172.21.15.69:0/3663210260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94140b2310 0x7f941413f790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f940c009e50 tx=0x7f940c02bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca5a3ee72c49c3d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 1 -- 172.21.15.69:0/3663210260 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c02caa0 con 0x7f94140b2310 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 10 monclient: dump: 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 1 -- 172.21.15.69:0/3663210260 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940c02cc00 con 0x7f94140b2310 2024-01-21T04:33:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 1 -- 172.21.15.69:0/3663210260 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940c0343f0 con 0x7f94140b2310 2024-01-21T04:33:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f9419338640 10 monclient: dump: 2024-01-21T04:33:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.747 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f941c59e640 5 monclient: authenticate success, global_id 4378 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f941c59e640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.742+0000 7f941c59e640 4 set_mon_vals no callback set 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.743+0000 7f941c59e640 10 monclient: shutdown 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.743+0000 7f941c59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.743+0000 7f941c59e640 1 -- 172.21.15.69:0/3663210260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94140b2310 msgr2=0x7f941413f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.743+0000 7f941c59e640 1 --2- 172.21.15.69:0/3663210260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94140b2310 0x7f941413f790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f940c009e50 tx=0x7f940c02bf00 comp rx=0 tx=0).stop 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.743+0000 7f941c59e640 1 -- 172.21.15.69:0/3663210260 shutdown_connections 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.743+0000 7f941c59e640 1 --2- 172.21.15.69:0/3663210260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94140b2310 0x7f941413f790 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.743+0000 7f941c59e640 1 --2- 172.21.15.69:0/3663210260 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94140a8b20 0x7f9414145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.743+0000 7f941c59e640 1 -- 172.21.15.69:0/3663210260 >> 172.21.15.69:0/3663210260 conn(0x7f9414066fb0 msgr2=0x7f94140b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.743+0000 7f941c59e640 1 -- 172.21.15.69:0/3663210260 shutdown_connections 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.744+0000 7f941c59e640 1 -- 172.21.15.69:0/3663210260 wait complete. 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.744+0000 7f941c59e640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.744+0000 7f941c59e640 10 monclient: monmap: 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:52.749 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:52.750 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:52.750 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.745+0000 7f941c59e640 1 librados: starting msgr at 2024-01-21T04:33:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.745+0000 7f941c59e640 1 librados: starting objecter 2024-01-21T04:33:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.745+0000 7f941c59e640 1 Processor -- start 2024-01-21T04:33:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.745+0000 7f941c59e640 1 -- start start 2024-01-21T04:33:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.745+0000 7f941c59e640 1 librados: setting wanted keys 2024-01-21T04:33:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.745+0000 7f941c59e640 1 librados: calling monclient init 2024-01-21T04:33:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.745+0000 7f941c59e640 10 monclient: init 2024-01-21T04:33:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94140a8b20 0x7f9414144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 10 monclient(hunting): picked mon.noname-a con 0x7f94140a8b20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94140a9d10 0x7f94141428d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 10 monclient(hunting): picked mon.noname-b con 0x7f94140a9d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94140b2310 0x7f94141d7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 10 monclient(hunting): picked mon.noname-c con 0x7f94140b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9414139160 con 0x7f94140a8b20 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9414137e70 con 0x7f94140a9d10 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9414141f30 con 0x7f94140b2310 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.746+0000 7f941c59e640 10 monclient(hunting): authenticate will time out at 1391.087976s 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.747+0000 7f9419b39640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94140a9d10 0x7f94141428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.747+0000 7f9419b39640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94140a9d10 0x7f94141428d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:60010/0 (socket says 172.21.15.69:60010) 2024-01-21T04:33:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.747+0000 7f9419b39640 1 -- 172.21.15.69:0/3770634122 learned_addr learned my addr 172.21.15.69:0/3770634122 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.747+0000 7f9419b39640 10 monclient(hunting): get_auth_request con 0x7f94140a9d10 auth_method 0 2024-01-21T04:33:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.747+0000 7f9419b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.747+0000 7f9419b39640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.747+0000 7f9419b39640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.748+0000 7f9419b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.748+0000 7f9419b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:52.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.748+0000 7f9419b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:52.756 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345928 got 85899345928 for osd.4 2024-01-21T04:33:52.756 DEBUG:teuthology.parallel:result is None 2024-01-21T04:33:52.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.751+0000 7f9419b39640 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2024-01-21T04:33:52.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.751+0000 7f9419b39640 1 -- 172.21.15.69:0/3770634122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94140b2310 msgr2=0x7f94141d7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.751+0000 7f9419b39640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94140b2310 0x7f94141d7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.751+0000 7f9419b39640 1 -- 172.21.15.69:0/3770634122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94140a8b20 msgr2=0x7f9414144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:52.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.751+0000 7f9419b39640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94140a8b20 0x7f9414144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:52.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.751+0000 7f9419b39640 10 monclient: _finish_hunting 0 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.751+0000 7f9419b39640 1 monclient: found mon.noname-b 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.751+0000 7f9419b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.751+0000 7f9419b39640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.751+0000 7f9419b39640 1 -- 172.21.15.69:0/3770634122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c009b30 con 0x7f94140a9d10 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.753+0000 7f9419b39640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94140a9d10 0x7f94141428d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f940400c400 tx=0x7f940400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.753+0000 7f940affd640 10 client.?.objecter ms_handle_connect 0x7f94140a9d10 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.753+0000 7f940affd640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 1 -- 172.21.15.69:0/3770634122 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94040052d0 con 0x7f94140a9d10 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 10 monclient: dump: 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.761 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 10 monclient: _finish_auth 0 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 1 -- 172.21.15.69:0/3770634122 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9404005430 con 0x7f94140a9d10 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f941c59e640 5 monclient: authenticate success, global_id 4381 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 1 -- 172.21.15.69:0/3770634122 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9404003830 con 0x7f94140a9d10 2024-01-21T04:33:52.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f940affd640 10 monclient: dump: 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.756+0000 7f9418b37640 4 set_mon_vals no callback set 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.757+0000 7f941c59e640 10 monclient: _renew_subs 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.757+0000 7f941c59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.757+0000 7f941c59e640 1 -- 172.21.15.69:0/3770634122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94141d7540 con 0x7f94140a9d10 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.757+0000 7f941c59e640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.757+0000 7f941c59e640 10 monclient: _renew_subs 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.757+0000 7f941c59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.757+0000 7f941c59e640 1 -- 172.21.15.69:0/3770634122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94141d7a00 con 0x7f94140a9d10 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.757+0000 7f941c59e640 1 librados: init done 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.758+0000 7f941c59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.758+0000 7f941c59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.758+0000 7f941c59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.758+0000 7f941c59e640 1 -- 172.21.15.69:0/3770634122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941404e970 con 0x7f94140a9d10 2024-01-21T04:33:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.777+0000 7f940affd640 1 -- 172.21.15.69:0/3770634122 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9404003990 con 0x7f94140a9d10 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.777+0000 7f47b515e640 10 monclient: get_monmap_and_config 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.777+0000 7f47b515e640 10 monclient: build_initial_monmap 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.777+0000 7f47b515e640 10 monclient: monmap: 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:52.778963+0000 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:52.778963+0000 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:52.780 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:52.780 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:52.780 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:52.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.778+0000 7f940affd640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f93f806c860 0x7f93f806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.778+0000 7f941a33a640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f93f806c860 0x7f93f806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.778+0000 7f941a33a640 10 monclient: get_auth_request con 0x7f93f806c860 auth_method 0 2024-01-21T04:33:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.778+0000 7f47b515e640 1 Processor -- start 2024-01-21T04:33:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.778+0000 7f47b515e640 1 -- start start 2024-01-21T04:33:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.778+0000 7f47b515e640 10 monclient: init 2024-01-21T04:33:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.779+0000 7f940affd640 10 client.4381.objecter ms_handle_connect 0x7f93f806c860 2024-01-21T04:33:52.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.779+0000 7f941a33a640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f93f806c860 0x7f93f806ed20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9410009710 tx=0x7f9410006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.779+0000 7f47b515e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.779+0000 7f47b515e640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-01-21T04:33:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.779+0000 7f47b515e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47b00aedd0 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.779+0000 7f47b515e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47b00af1b0 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.779+0000 7f47b515e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f47b00ac8f0 0x7f47b014d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.779+0000 7f47b515e640 10 monclient(hunting): picked mon.noname-a con 0x7f47b00ac8f0 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.780+0000 7f47b515e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b006ef00 con 0x7f47b00aedd0 2024-01-21T04:33:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.780+0000 7f47b515e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b006f1e0 con 0x7f47b00af1b0 2024-01-21T04:33:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.780+0000 7f47b515e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.780+0000 7f47b515e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b00acd00 con 0x7f47b00ac8f0 2024-01-21T04:33:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.780+0000 7f47b515e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.780+0000 7f47b515e640 10 monclient(hunting): authenticate will time out at 1391.121408s 2024-01-21T04:33:52.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.814+0000 7fdd4877d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T04:33:52.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.814+0000 7fdd4877d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T04:33:52.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.814+0000 7fdd4877d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:52.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.814+0000 7fdd4877d640 1 -- 172.21.15.69:0/3672666549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fdd400abb70 con 0x7fdd400720e0 2024-01-21T04:33:52.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.867+0000 7fc234ff9640 1 -- 172.21.15.69:0/365757925 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc2300547a0 con 0x7fc238137a40 2024-01-21T04:33:52.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.867+0000 7fc234ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:52.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.867+0000 7fc234ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:52.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.971+0000 7fdc9e1b6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T04:33:52.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.971+0000 7fdc9e1b6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T04:33:52.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.971+0000 7fdc9e1b6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:52.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:52.971+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2839231803 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdc78005600 con 0x7fdc980a8b20 2024-01-21T04:33:53.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.004+0000 7f940affd640 1 -- 172.21.15.69:0/3770634122 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9404058370 con 0x7f94140a9d10 2024-01-21T04:33:53.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.004+0000 7f940affd640 10 client.4381.objecter ms_dispatch 0x7f9414000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:53.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.004+0000 7f940affd640 3 client.4381.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:53.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.004+0000 7f940affd640 3 client.4381.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:53.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.004+0000 7f940affd640 20 client.4381.objecter dump_active .. 0 homeless 2024-01-21T04:33:53.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.015+0000 7f940affd640 1 -- 172.21.15.69:0/3770634122 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f9404053fb0 con 0x7f94140a9d10 2024-01-21T04:33:53.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.015+0000 7f940affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:53.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.015+0000 7f940affd640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:53.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.016+0000 7f47ae59c640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f47b00af1b0 0x7f47b0071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:54070/0 (socket says 172.21.15.69:54070) 2024-01-21T04:33:53.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.016+0000 7f47ae59c640 1 -- 172.21.15.69:0/1442824395 learned_addr learned my addr 172.21.15.69:0/1442824395 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:53.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.018+0000 7f6b2ffff640 1 -- 172.21.15.69:0/2581982995 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6b3c076480 con 0x7f6b480a9d10 2024-01-21T04:33:53.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.018+0000 7f6b2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:53.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.018+0000 7f6b2ffff640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:53.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.019+0000 7f4231efa640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 0x7f422c0b1070 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f422800c7a0 tx=0x7f422800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:53.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.019+0000 7f42227fc640 10 client.?.objecter ms_handle_connect 0x7f4218004e20 2024-01-21T04:33:53.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.019+0000 7f42227fc640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:53.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.021+0000 7fdc7ffff640 1 -- 172.21.15.69:0/2839231803 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdc88076a10 con 0x7fdc980a8b20 2024-01-21T04:33:53.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.021+0000 7fdc7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-01-21T04:33:53.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.021+0000 7fdc7ffff640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:53.022 INFO:teuthology.orchestra.run.smithi069.stdout:64424509446 2024-01-21T04:33:53.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.023+0000 7fdc9e1b6640 10 librados: watch_flush enter 2024-01-21T04:33:53.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.023+0000 7fdc9e1b6640 10 librados: watch_flush exit 2024-01-21T04:33:53.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.023+0000 7fdc9e1b6640 20 client.4375.objecter shutdown clearing up homeless session... 2024-01-21T04:33:53.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.023+0000 7fdc9e1b6640 10 client.4375.objecter successfully canceled tick 2024-01-21T04:33:53.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.023+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2839231803 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdc8406c890 msgr2=0x7fdc8406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.023+0000 7fdc9e1b6640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdc8406c890 0x7fdc8406ed50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdc981422c0 tx=0x7fdc8c007990 comp rx=0 tx=0).stop 2024-01-21T04:33:53.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.023+0000 7fdc9e1b6640 10 monclient: shutdown 2024-01-21T04:33:53.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.023+0000 7fdc9e1b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:53.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.023+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2839231803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc980a8b20 msgr2=0x7fdc98140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.023+0000 7fdc9e1b6640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc980a8b20 0x7fdc98140f40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdc880090b0 tx=0x7fdc880046c0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.024+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2839231803 shutdown_connections 2024-01-21T04:33:53.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.024+0000 7fdc9e1b6640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdc8406c890 0x7fdc8406ed50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.024+0000 7fdc9e1b6640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdc980b2310 0x7fdc98132620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.024+0000 7fdc9e1b6640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc980a9d10 0x7fdc981405d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.024+0000 7fdc9e1b6640 1 --2- 172.21.15.69:0/2839231803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdc980a8b20 0x7fdc98140f40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.024+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2839231803 >> 172.21.15.69:0/2839231803 conn(0x7fdc98066fb0 msgr2=0x7fdc980b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:53.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.024+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2839231803 shutdown_connections 2024-01-21T04:33:53.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.024+0000 7fdc9e1b6640 1 -- 172.21.15.69:0/2839231803 wait complete. 2024-01-21T04:33:53.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.024+0000 7fdc9e1b6640 1 librados: shutdown 2024-01-21T04:33:53.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.0 2024-01-21T04:33:53.037 DEBUG:teuthology.parallel:result is None 2024-01-21T04:33:53.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.036+0000 7f47af59e640 1 --2- 172.21.15.69:0/1442824395 >> v2:172.21.15.69:3300/0 conn(0x7f47b00ac8f0 0x7f47b014d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:53.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.038+0000 7f47add9b640 1 -- 172.21.15.69:0/1442824395 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234920138 0 0) 0x7f47b006ef00 con 0x7f47b00aedd0 2024-01-21T04:33:53.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.038+0000 7f47add9b640 10 monclient(hunting): _negotiate 2024-01-21T04:33:53.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.038+0000 7f47add9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:53.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.038+0000 7f47add9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:53.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.038+0000 7f47add9b640 10 monclient(hunting): my global_id is 4365 2024-01-21T04:33:53.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.038+0000 7f47add9b640 1 -- 172.21.15.69:0/1442824395 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4798003520 con 0x7f47b00aedd0 2024-01-21T04:33:53.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.038+0000 7f47af59e640 10 monclient(hunting): get_auth_request con 0x7f47b00ac8f0 auth_method 0 2024-01-21T04:33:53.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.038+0000 7f47af59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:53.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.038+0000 7f47af59e640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:53.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.038+0000 7f47af59e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:53.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.039+0000 7f47af59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:53.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.039+0000 7f47af59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:53.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.039+0000 7f47af59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:53.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.042+0000 7f47af59e640 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2024-01-21T04:33:53.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.042+0000 7f47af59e640 1 -- 172.21.15.69:0/1442824395 >> v1:172.21.15.144:6789/0 conn(0x7f47b00af1b0 legacy=0x7f47b0071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.042+0000 7f47af59e640 1 -- 172.21.15.69:0/1442824395 >> v1:172.21.15.69:6789/0 conn(0x7f47b00aedd0 legacy=0x7f47b0142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.042+0000 7f47af59e640 10 monclient: _finish_hunting 0 2024-01-21T04:33:53.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.042+0000 7f47af59e640 1 monclient: found mon.noname-a 2024-01-21T04:33:53.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.042+0000 7f47af59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:53.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.042+0000 7f47af59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:53.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.042+0000 7f47af59e640 1 -- 172.21.15.69:0/1442824395 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b006f4c0 con 0x7f47b00ac8f0 2024-01-21T04:33:53.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.053+0000 7f47af59e640 1 --2- 172.21.15.69:0/1442824395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47b00ac8f0 0x7f47b014d880 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009d90 tx=0x7f47a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=264d4df8a13f5deb server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:53.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 1 -- 172.21.15.69:0/935342505 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42280074e0 con 0x7f4218004e20 2024-01-21T04:33:53.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:53.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:53.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 10 monclient: dump: 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 10 monclient: _finish_auth 0 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 10 monclient: _check_auth_tickets 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:53.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 1 -- 172.21.15.69:0/935342505 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4228007640 con 0x7f4218004e20 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f423415e640 5 monclient: authenticate success, global_id 4362 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f423415e640 10 monclient: _renew_subs 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f423415e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f423415e640 1 -- 172.21.15.69:0/935342505 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c0ab980 con 0x7f4218004e20 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42237fe640 4 set_mon_vals no callback set 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f423415e640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f423415e640 10 monclient: _renew_subs 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f423415e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f423415e640 1 -- 172.21.15.69:0/935342505 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c1cb710 con 0x7f4218004e20 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f423415e640 1 librados: init done 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 1 -- 172.21.15.69:0/935342505 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228004a20 con 0x7f4218004e20 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:53.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.058+0000 7f42227fc640 10 monclient: dump: 2024-01-21T04:33:53.062 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:53.062 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:53.062 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.062 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.062 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:53.062 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:53.062 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:53.062 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:53.062 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:53.062 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:53.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.059+0000 7f423415e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:53.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.059+0000 7f423415e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:53.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.059+0000 7f423415e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:53.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.059+0000 7f423415e640 1 -- 172.21.15.69:0/935342505 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c04e970 con 0x7f4218004e20 2024-01-21T04:33:53.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.089+0000 7fc23e705640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T04:33:53.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.089+0000 7fc23e705640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T04:33:53.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.089+0000 7fc23e705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T04:33:53.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.089+0000 7fc23e705640 1 -- 172.21.15.69:0/365757925 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc2381402a0 con 0x7fc238137a40 2024-01-21T04:33:53.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.121+0000 7fc234ff9640 1 -- 172.21.15.69:0/365757925 <== mon.2 v2:172.21.15.69:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc2381402a0 con 0x7fc238137a40 2024-01-21T04:33:53.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.121+0000 7fc234ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-01-21T04:33:53.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.121+0000 7fc234ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:53.122 INFO:teuthology.orchestra.run.smithi069.stdout:85899345928 2024-01-21T04:33:53.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.124+0000 7fc23e705640 10 librados: watch_flush enter 2024-01-21T04:33:53.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 10 librados: watch_flush exit 2024-01-21T04:33:53.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 20 client.4316.objecter shutdown clearing up homeless session... 2024-01-21T04:33:53.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 10 client.4316.objecter successfully canceled tick 2024-01-21T04:33:53.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 -- 172.21.15.69:0/365757925 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc22406c860 msgr2=0x7fc22406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc22406c860 0x7fc22406ed20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc228001560 tx=0x7fc228007990 comp rx=0 tx=0).stop 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 10 monclient: shutdown 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 -- 172.21.15.69:0/365757925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc238137a40 msgr2=0x7fc238140ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc238137a40 0x7fc238140ea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc23000b9e0 tx=0x7fc23000bea0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 -- 172.21.15.69:0/365757925 shutdown_connections 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc22406c860 0x7fc22406ed20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc238137a40 0x7fc238140ea0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc2380b2310 0x7fc238140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 --2- 172.21.15.69:0/365757925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2380a8b20 0x7fc238137130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 -- 172.21.15.69:0/365757925 >> 172.21.15.69:0/365757925 conn(0x7fc238066fb0 msgr2=0x7fc2380b6490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 -- 172.21.15.69:0/365757925 shutdown_connections 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 -- 172.21.15.69:0/365757925 wait complete. 2024-01-21T04:33:53.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.125+0000 7fc23e705640 1 librados: shutdown 2024-01-21T04:33:53.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345928 got 85899345928 for osd.6 2024-01-21T04:33:53.141 DEBUG:teuthology.parallel:result is None 2024-01-21T04:33:53.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.229+0000 7f6b4e0a2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T04:33:53.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.229+0000 7f6b4e0a2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T04:33:53.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.229+0000 7f6b4e0a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:53.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.229+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2581982995 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f6b48143190 con 0x7f6b480a9d10 2024-01-21T04:33:53.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.238+0000 7f941c59e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T04:33:53.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.238+0000 7f941c59e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T04:33:53.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.238+0000 7f941c59e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:53.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.238+0000 7f941c59e640 1 -- 172.21.15.69:0/3770634122 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f94141420f0 con 0x7f94140a9d10 2024-01-21T04:33:53.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.242+0000 7f940affd640 1 -- 172.21.15.69:0/3770634122 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9404017020 con 0x7f94140a9d10 2024-01-21T04:33:53.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.242+0000 7f940affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-01-21T04:33:53.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.242+0000 7f940affd640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:53.243 INFO:teuthology.orchestra.run.smithi069.stdout:64424509448 2024-01-21T04:33:53.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.243+0000 7f941c59e640 10 librados: watch_flush enter 2024-01-21T04:33:53.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.243+0000 7f941c59e640 10 librados: watch_flush exit 2024-01-21T04:33:53.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.243+0000 7f941c59e640 20 client.4381.objecter shutdown clearing up homeless session... 2024-01-21T04:33:53.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.243+0000 7f941c59e640 10 client.4381.objecter successfully canceled tick 2024-01-21T04:33:53.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 -- 172.21.15.69:0/3770634122 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f93f806c860 msgr2=0x7f93f806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f93f806c860 0x7f93f806ed20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9410009710 tx=0x7f9410006c90 comp rx=0 tx=0).stop 2024-01-21T04:33:53.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 10 monclient: shutdown 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 -- 172.21.15.69:0/3770634122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94140a9d10 msgr2=0x7f94141428d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94140a9d10 0x7f94141428d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f940400c400 tx=0x7f940400c8c0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 -- 172.21.15.69:0/3770634122 shutdown_connections 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f93f806c860 0x7f93f806ed20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f94140b2310 0x7f94141d7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94140a9d10 0x7f94141428d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 --2- 172.21.15.69:0/3770634122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f94140a8b20 0x7f9414144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 -- 172.21.15.69:0/3770634122 >> 172.21.15.69:0/3770634122 conn(0x7f9414066fb0 msgr2=0x7f94140b61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 -- 172.21.15.69:0/3770634122 shutdown_connections 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 -- 172.21.15.69:0/3770634122 wait complete. 2024-01-21T04:33:53.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.244+0000 7f941c59e640 1 librados: shutdown 2024-01-21T04:33:53.255 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.2 2024-01-21T04:33:53.256 DEBUG:teuthology.parallel:result is None 2024-01-21T04:33:53.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.386+0000 7fdd36ffd640 1 -- 172.21.15.69:0/3672666549 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdd3c020020 con 0x7fdd400720e0 2024-01-21T04:33:53.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.386+0000 7fdd36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-01-21T04:33:53.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.386+0000 7fdd36ffd640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:53.388 INFO:teuthology.orchestra.run.smithi069.stdout:55834574854 2024-01-21T04:33:53.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 10 librados: watch_flush enter 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 10 librados: watch_flush exit 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 20 client.4350.objecter shutdown clearing up homeless session... 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 10 client.4350.objecter successfully canceled tick 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 -- 172.21.15.69:0/3672666549 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdd2406c8b0 msgr2=0x7fdd2406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdd2406c8b0 0x7fdd2406ed70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdd400acdb0 tx=0x7fdd300091c0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 10 monclient: shutdown 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 -- 172.21.15.69:0/3672666549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd400720e0 msgr2=0x7fdd40141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd400720e0 0x7fdd40141690 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00e930 tx=0x7fdd3c00edf0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 -- 172.21.15.69:0/3672666549 shutdown_connections 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdd2406c8b0 0x7fdd2406ed70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd4013f520 0x7fdd400a97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd400a8750 0x7fdd400ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 --2- 172.21.15.69:0/3672666549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd400720e0 0x7fdd40141690 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 -- 172.21.15.69:0/3672666549 >> 172.21.15.69:0/3672666549 conn(0x7fdd40066f00 msgr2=0x7fdd4013bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:53.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 -- 172.21.15.69:0/3672666549 shutdown_connections 2024-01-21T04:33:53.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 -- 172.21.15.69:0/3672666549 wait complete. 2024-01-21T04:33:53.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.389+0000 7fdd4877d640 1 librados: shutdown 2024-01-21T04:33:53.403 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-01-21T04:33:53.403 DEBUG:teuthology.parallel:result is None 2024-01-21T04:33:53.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 1 -- 172.21.15.69:0/1442824395 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a802cc80 con 0x7f47b00ac8f0 2024-01-21T04:33:53.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:53.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:53.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 10 monclient: dump: 2024-01-21T04:33:53.410 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:53.410 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:53.410 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 10 monclient: _finish_auth 0 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 10 monclient: _check_auth_tickets 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 1 -- 172.21.15.69:0/1442824395 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a802cde0 con 0x7f47b00ac8f0 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47b515e640 5 monclient: authenticate success, global_id 4368 2024-01-21T04:33:53.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47b515e640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47b515e640 4 set_mon_vals no callback set 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 1 -- 172.21.15.69:0/1442824395 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a8034660 con 0x7f47b00ac8f0 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.409+0000 7f47add9b640 10 monclient: dump: 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:53.412 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:53.413 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:53.413 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:53.413 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:53.413 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:53.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.410+0000 7f47b515e640 10 monclient: shutdown 2024-01-21T04:33:53.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.410+0000 7f47b515e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:53.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.410+0000 7f47b515e640 1 -- 172.21.15.69:0/1442824395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47b00ac8f0 msgr2=0x7f47b014d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.410+0000 7f47b515e640 1 --2- 172.21.15.69:0/1442824395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47b00ac8f0 0x7f47b014d880 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009d90 tx=0x7f47a802bed0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.410+0000 7f47b515e640 1 -- 172.21.15.69:0/1442824395 shutdown_connections 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.410+0000 7f47b515e640 1 --2- 172.21.15.69:0/1442824395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47b00ac8f0 0x7f47b014d880 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.410+0000 7f47b515e640 1 -- 172.21.15.69:0/1442824395 >> 172.21.15.69:0/1442824395 conn(0x7f47b00b3fe0 msgr2=0x7f47b00b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.410+0000 7f47b515e640 1 -- 172.21.15.69:0/1442824395 shutdown_connections 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.410+0000 7f47b515e640 1 -- 172.21.15.69:0/1442824395 wait complete. 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.411+0000 7f47b515e640 10 monclient: build_initial_monmap 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.411+0000 7f47b515e640 10 monclient: monmap: 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:53.416 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.411+0000 7f47b515e640 1 librados: starting msgr at 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.411+0000 7f47b515e640 1 librados: starting objecter 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.411+0000 7f47b515e640 1 Processor -- start 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.411+0000 7f47b515e640 1 -- start start 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.411+0000 7f47b515e640 1 librados: setting wanted keys 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.411+0000 7f47b515e640 1 librados: calling monclient init 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.411+0000 7f47b515e640 10 monclient: init 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:33:53.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47b00ac8f0 0x7f47b013e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 10 monclient(hunting): picked mon.noname-a con 0x7f47b00ac8f0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47b00aedd0 0x7f47b0070c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 10 monclient(hunting): picked mon.noname-c con 0x7f47b00aedd0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b00af1b0 0x7f47b013b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 10 monclient(hunting): picked mon.noname-b con 0x7f47b00af1b0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b013ec70 con 0x7f47b00ac8f0 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b013f190 con 0x7f47b00af1b0 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0071a50 con 0x7f47b00aedd0 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47b515e640 10 monclient(hunting): authenticate will time out at 1391.753393s 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47ae59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47b00aedd0 0x7f47b0070c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:53.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.412+0000 7f47af59e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b00af1b0 0x7f47b013b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47ae59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47b00aedd0 0x7f47b0070c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:47090/0 (socket says 172.21.15.69:47090) 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47ae59c640 1 -- 172.21.15.69:0/1151680659 learned_addr learned my addr 172.21.15.69:0/1151680659 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47ae59c640 10 monclient(hunting): get_auth_request con 0x7f47b00aedd0 auth_method 0 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47ae59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47af59e640 10 monclient(hunting): get_auth_request con 0x7f47b00af1b0 auth_method 0 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47af59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47af59e640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47af59e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:53.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47af59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:53.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47af59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:53.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.413+0000 7f47af59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:53.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.416+0000 7f47af59e640 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.416+0000 7f47af59e640 1 -- 172.21.15.69:0/1151680659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47b00aedd0 msgr2=0x7f47b0070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.416+0000 7f47af59e640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47b00aedd0 0x7f47b0070c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.416+0000 7f47af59e640 1 -- 172.21.15.69:0/1151680659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47b00ac8f0 msgr2=0x7f47b013e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.416+0000 7f47af59e640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47b00ac8f0 0x7f47b013e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.416+0000 7f47af59e640 10 monclient: _finish_hunting 0 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.416+0000 7f47af59e640 1 monclient: found mon.noname-b 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.416+0000 7f47af59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.416+0000 7f47af59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.416+0000 7f47af59e640 1 -- 172.21.15.69:0/1151680659 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a8009a70 con 0x7f47b00af1b0 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.418+0000 7f47af59e640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b00af1b0 0x7f47b013b660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f47a8033000 tx=0x7f47a8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.418+0000 7f47977fe640 10 client.?.objecter ms_handle_connect 0x7f47b00af1b0 2024-01-21T04:33:53.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.418+0000 7f47977fe640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:53.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47977fe640 1 -- 172.21.15.69:0/1151680659 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a802cc80 con 0x7f47b00af1b0 2024-01-21T04:33:53.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:53.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:53.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47977fe640 10 monclient: dump: 2024-01-21T04:33:53.423 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:53.423 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:53.423 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.423 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.423 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:53.423 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47977fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47977fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47b515e640 5 monclient: authenticate success, global_id 4384 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47b515e640 10 monclient: _renew_subs 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47b515e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47b515e640 1 -- 172.21.15.69:0/1151680659 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b013e890 con 0x7f47b00af1b0 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47b515e640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:53.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47b515e640 10 monclient: _renew_subs 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47b515e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47b515e640 1 -- 172.21.15.69:0/1151680659 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b013bd80 con 0x7f47b00af1b0 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.420+0000 7f47b515e640 1 librados: init done 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.422+0000 7f47b515e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.422+0000 7f47b515e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.422+0000 7f47b515e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.422+0000 7f47b515e640 1 -- 172.21.15.69:0/1151680659 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b0142ce0 con 0x7f47b00af1b0 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.422+0000 7f47977fe640 1 -- 172.21.15.69:0/1151680659 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a8053480 con 0x7f47b00af1b0 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.422+0000 7f47977fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.422+0000 7f47977fe640 1 -- 172.21.15.69:0/1151680659 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a8053810 con 0x7f47b00af1b0 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.422+0000 7f47977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.422+0000 7f47977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.422+0000 7f47977fe640 10 monclient: dump: 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:53.425 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:53.426 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.426 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:53.426 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:53.426 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:53.426 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:53.426 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:53.426 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:53.426 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:53.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.422+0000 7f47ad59a640 4 set_mon_vals no callback set 2024-01-21T04:33:53.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.423+0000 7f42227fc640 1 -- 172.21.15.69:0/935342505 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f422800a5e0 con 0x7f4218004e20 2024-01-21T04:33:53.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.423+0000 7f42227fc640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f421006c8b0 0x7f421006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:53.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.424+0000 7f42316f9640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f421006c8b0 0x7f421006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:53.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.424+0000 7f42316f9640 10 monclient: get_auth_request con 0x7f421006c8b0 auth_method 0 2024-01-21T04:33:53.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.424+0000 7f42227fc640 10 client.4362.objecter ms_handle_connect 0x7f421006c8b0 2024-01-21T04:33:53.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.424+0000 7f42316f9640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f421006c8b0 0x7f421006ed70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f422c1431d0 tx=0x7f421c006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.438+0000 7f42227fc640 1 -- 172.21.15.69:0/935342505 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f4228058630 con 0x7f4218004e20 2024-01-21T04:33:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.438+0000 7f42227fc640 10 client.4362.objecter ms_dispatch 0x7f422c000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.438+0000 7f42227fc640 3 client.4362.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:53.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.438+0000 7f42227fc640 3 client.4362.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:53.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.438+0000 7f42227fc640 20 client.4362.objecter dump_active .. 0 homeless 2024-01-21T04:33:53.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.438+0000 7f42227fc640 1 -- 172.21.15.69:0/935342505 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4228054270 con 0x7f4218004e20 2024-01-21T04:33:53.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.438+0000 7f42227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:53.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.438+0000 7f42227fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:53.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.440+0000 7f6b2ffff640 1 -- 172.21.15.69:0/2581982995 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6b48143190 con 0x7f6b480a9d10 2024-01-21T04:33:53.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.440+0000 7f6b2ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-01-21T04:33:53.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.440+0000 7f6b2ffff640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:53.441 INFO:teuthology.orchestra.run.smithi069.stdout:64424509448 2024-01-21T04:33:53.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 10 librados: watch_flush enter 2024-01-21T04:33:53.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 10 librados: watch_flush exit 2024-01-21T04:33:53.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 20 client.4359.objecter shutdown clearing up homeless session... 2024-01-21T04:33:53.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 10 client.4359.objecter successfully canceled tick 2024-01-21T04:33:53.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2581982995 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b3406c860 msgr2=0x7f6b3406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b3406c860 0x7f6b3406ed20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009650 tx=0x7f6b38006ba0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 10 monclient: shutdown 2024-01-21T04:33:53.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:53.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2581982995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b480a9d10 msgr2=0x7f6b4813a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b480a9d10 0x7f6b4813a6b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0090b0 tx=0x7f6b3c0046c0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2581982995 shutdown_connections 2024-01-21T04:33:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6b3406c860 0x7f6b3406ed20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6b480b2310 0x7f6b4813abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b480a9d10 0x7f6b4813a6b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 1 --2- 172.21.15.69:0/2581982995 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6b480a8b20 0x7f6b48140bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.442+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2581982995 >> 172.21.15.69:0/2581982995 conn(0x7f6b48066fb0 msgr2=0x7f6b480b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.443+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2581982995 shutdown_connections 2024-01-21T04:33:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.443+0000 7f6b4e0a2640 1 -- 172.21.15.69:0/2581982995 wait complete. 2024-01-21T04:33:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.443+0000 7f6b4e0a2640 1 librados: shutdown 2024-01-21T04:33:53.456 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.7 2024-01-21T04:33:53.456 DEBUG:teuthology.parallel:result is None 2024-01-21T04:33:53.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.616+0000 7f423415e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T04:33:53.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.616+0000 7f423415e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T04:33:53.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.616+0000 7f423415e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:33:53.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.616+0000 7f423415e640 1 -- 172.21.15.69:0/935342505 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f422c1cb8a0 con 0x7f4218004e20 2024-01-21T04:33:53.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.619+0000 7f42227fc640 1 -- 172.21.15.69:0/935342505 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4228018070 con 0x7f4218004e20 2024-01-21T04:33:53.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.619+0000 7f42227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-01-21T04:33:53.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.619+0000 7f42227fc640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:53.621 INFO:teuthology.orchestra.run.smithi069.stdout:64424509446 2024-01-21T04:33:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.621+0000 7f423415e640 10 librados: watch_flush enter 2024-01-21T04:33:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.621+0000 7f423415e640 10 librados: watch_flush exit 2024-01-21T04:33:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.621+0000 7f423415e640 20 client.4362.objecter shutdown clearing up homeless session... 2024-01-21T04:33:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.621+0000 7f423415e640 10 client.4362.objecter successfully canceled tick 2024-01-21T04:33:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.621+0000 7f423415e640 1 -- 172.21.15.69:0/935342505 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f421006c8b0 msgr2=0x7f421006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.621+0000 7f423415e640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f421006c8b0 0x7f421006ed70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f422c1431d0 tx=0x7f421c006c90 comp rx=0 tx=0).stop 2024-01-21T04:33:53.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.621+0000 7f423415e640 10 monclient: shutdown 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.621+0000 7f423415e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.621+0000 7f423415e640 1 -- 172.21.15.69:0/935342505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 msgr2=0x7f422c0b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.621+0000 7f423415e640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 0x7f422c0b1070 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f422800c7a0 tx=0x7f422800cc60 comp rx=0 tx=0).stop 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.622+0000 7f423415e640 1 -- 172.21.15.69:0/935342505 shutdown_connections 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.622+0000 7f423415e640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f421006c8b0 0x7f421006ed70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.622+0000 7f423415e640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f422c1cb2b0 0x7f422c1da690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.622+0000 7f423415e640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f422c1cd5c0 0x7f422c1ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.622+0000 7f423415e640 1 --2- 172.21.15.69:0/935342505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4218004e20 0x7f422c0b1070 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.622+0000 7f423415e640 1 -- 172.21.15.69:0/935342505 >> 172.21.15.69:0/935342505 conn(0x7f422c066fb0 msgr2=0x7f422c13e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.622+0000 7f423415e640 1 -- 172.21.15.69:0/935342505 shutdown_connections 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.622+0000 7f423415e640 1 -- 172.21.15.69:0/935342505 wait complete. 2024-01-21T04:33:53.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.622+0000 7f423415e640 1 librados: shutdown 2024-01-21T04:33:53.633 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.1 2024-01-21T04:33:53.633 DEBUG:teuthology.parallel:result is None 2024-01-21T04:33:53.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.667+0000 7f47977fe640 1 -- 172.21.15.69:0/1151680659 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f47a803b050 con 0x7f47b00af1b0 2024-01-21T04:33:53.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.668+0000 7f47977fe640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f479c06c860 0x7f479c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:53.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.668+0000 7f47aed9d640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f479c06c860 0x7f479c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:53.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.668+0000 7f47aed9d640 10 monclient: get_auth_request con 0x7f479c06c860 auth_method 0 2024-01-21T04:33:53.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.668+0000 7f47aed9d640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f479c06c860 0x7f479c06ed20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f47a00010e0 tx=0x7f47a0007910 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:53.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.668+0000 7f47977fe640 10 client.4384.objecter ms_handle_connect 0x7f479c06c860 2024-01-21T04:33:53.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.669+0000 7f47977fe640 1 -- 172.21.15.69:0/1151680659 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f47a807ad50 con 0x7f47b00af1b0 2024-01-21T04:33:53.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.670+0000 7f47977fe640 10 client.4384.objecter ms_dispatch 0x7f47b0000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:53.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.670+0000 7f47977fe640 3 client.4384.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:53.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.670+0000 7f47977fe640 3 client.4384.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:53.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.670+0000 7f47977fe640 20 client.4384.objecter dump_active .. 0 homeless 2024-01-21T04:33:53.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.679+0000 7f47977fe640 1 -- 172.21.15.69:0/1151680659 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f47a8077090 con 0x7f47b00af1b0 2024-01-21T04:33:53.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.679+0000 7f47977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:53.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.679+0000 7f47977fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:53.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.849+0000 7f47b515e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T04:33:53.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.849+0000 7f47b515e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T04:33:53.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.849+0000 7f47b515e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:53.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.849+0000 7f47b515e640 1 -- 172.21.15.69:0/1151680659 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f47b004e970 con 0x7f47b00af1b0 2024-01-21T04:33:53.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.852+0000 7f47977fe640 1 -- 172.21.15.69:0/1151680659 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f47a804e090 con 0x7f47b00af1b0 2024-01-21T04:33:53.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.852+0000 7f47977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-01-21T04:33:53.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.852+0000 7f47977fe640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:53.853 INFO:teuthology.orchestra.run.smithi069.stdout:90194313226 2024-01-21T04:33:53.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 10 librados: watch_flush enter 2024-01-21T04:33:53.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 10 librados: watch_flush exit 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 20 client.4384.objecter shutdown clearing up homeless session... 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 10 client.4384.objecter successfully canceled tick 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 -- 172.21.15.69:0/1151680659 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f479c06c860 msgr2=0x7f479c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f479c06c860 0x7f479c06ed20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f47a00010e0 tx=0x7f47a0007910 comp rx=0 tx=0).stop 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 10 monclient: shutdown 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 -- 172.21.15.69:0/1151680659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b00af1b0 msgr2=0x7f47b013b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b00af1b0 0x7f47b013b660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f47a8033000 tx=0x7f47a8004a40 comp rx=0 tx=0).stop 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 -- 172.21.15.69:0/1151680659 shutdown_connections 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f479c06c860 0x7f479c06ed20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47b00af1b0 0x7f47b013b660 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f47b00aedd0 0x7f47b0070c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 --2- 172.21.15.69:0/1151680659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47b00ac8f0 0x7f47b013e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:53.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 -- 172.21.15.69:0/1151680659 >> 172.21.15.69:0/1151680659 conn(0x7f47b00b3fe0 msgr2=0x7f47b00b19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:53.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 -- 172.21.15.69:0/1151680659 shutdown_connections 2024-01-21T04:33:53.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 -- 172.21.15.69:0/1151680659 wait complete. 2024-01-21T04:33:53.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:53.854+0000 7f47b515e640 1 librados: shutdown 2024-01-21T04:33:53.865 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313226 got 90194313226 for osd.3 2024-01-21T04:33:53.866 DEBUG:teuthology.parallel:result is None 2024-01-21T04:33:53.866 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-21T04:33:53.866 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T04:33:54.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.004+0000 7f12bf141640 10 monclient: get_monmap_and_config 2024-01-21T04:33:54.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.004+0000 7f12bf141640 10 monclient: build_initial_monmap 2024-01-21T04:33:54.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.004+0000 7f12bf141640 10 monclient: monmap: 2024-01-21T04:33:54.005 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:54.005 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:54.005 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:54.005423+0000 2024-01-21T04:33:54.006 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:54.005423+0000 2024-01-21T04:33:54.006 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:54.006 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.006 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:54.006 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:54.006 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:54.006 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:54.006 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:54.006 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 1 Processor -- start 2024-01-21T04:33:54.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 1 -- start start 2024-01-21T04:33:54.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 10 monclient: init 2024-01-21T04:33:54.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:54.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-01-21T04:33:54.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12b80b12d0 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:54.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12b8142b90 0x7f12b8141f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:54.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 10 monclient(hunting): picked mon.noname-b con 0x7f12b8142b90 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:54.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12b80b16b0 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:54.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b80b1b40 con 0x7f12b80b12d0 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b8142670 con 0x7f12b80b16b0 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b806d090 con 0x7f12b8142b90 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.005+0000 7f12bf141640 10 monclient(hunting): authenticate will time out at 1392.347049s 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.006+0000 7f12bcedd640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f12b80b12d0 0x7f12b806c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59000/0 (socket says 172.21.15.69:59000) 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.006+0000 7f12bcedd640 1 -- 172.21.15.69:0/1833220035 learned_addr learned my addr 172.21.15.69:0/1833220035 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.006+0000 7f12b7fff640 1 --2- 172.21.15.69:0/1833220035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12b8142b90 0x7f12b8141f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.006+0000 7f12b7fff640 10 monclient(hunting): get_auth_request con 0x7f12b8142b90 auth_method 0 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.006+0000 7f12b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.006+0000 7f12b7fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.006+0000 7f12b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:54.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.007+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:54.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.007+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:54.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.007+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 1 -- 172.21.15.69:0/1833220035 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594659170 0 0) 0x7f12b8142670 con 0x7f12b80b16b0 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 10 monclient(hunting): _negotiate 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 10 monclient(hunting): my global_id is 4387 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 1 -- 172.21.15.69:0/1833220035 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a8003520 con 0x7f12b80b16b0 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 1 -- 172.21.15.69:0/1833220035 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592344715 0 0) 0x7f12b80b1b40 con 0x7f12b80b12d0 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 10 monclient(hunting): _negotiate 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 10 monclient(hunting): my global_id is 4371 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b77fe640 1 -- 172.21.15.69:0/1833220035 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b8142670 con 0x7f12b80b12d0 2024-01-21T04:33:54.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b7fff640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-01-21T04:33:54.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b7fff640 1 -- 172.21.15.69:0/1833220035 >> v1:172.21.15.144:6789/0 conn(0x7f12b80b16b0 legacy=0x7f12b8145450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b7fff640 1 -- 172.21.15.69:0/1833220035 >> v1:172.21.15.69:6789/0 conn(0x7f12b80b12d0 legacy=0x7f12b806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b7fff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:54.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b7fff640 1 monclient: found mon.noname-b 2024-01-21T04:33:54.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:54.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T04:33:54.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.008+0000 7f12b7fff640 1 -- 172.21.15.69:0/1833220035 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b8070380 con 0x7f12b8142b90 2024-01-21T04:33:54.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.010+0000 7f12b7fff640 1 --2- 172.21.15.69:0/1833220035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12b8142b90 0x7f12b8141f30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f12a40097d0 tx=0x7f12a402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efb5ef38e405197f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:54.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 1 -- 172.21.15.69:0/1833220035 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4031800 con 0x7f12b8142b90 2024-01-21T04:33:54.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:54.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:54.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 10 monclient: dump: 2024-01-21T04:33:54.016 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:54.017 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:54.017 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.017 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.017 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:54.017 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.017 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:54.017 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:54.017 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:54.017 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:54.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:54.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:54.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 1 -- 172.21.15.69:0/1833220035 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a4031960 con 0x7f12b8142b90 2024-01-21T04:33:54.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:54.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 1 -- 172.21.15.69:0/1833220035 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4031cf0 con 0x7f12b8142b90 2024-01-21T04:33:54.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:54.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T04:33:54.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12b77fe640 10 monclient: dump: 2024-01-21T04:33:54.018 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:54.018 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:54.019 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.019 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.019 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:54.019 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.019 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:54.019 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:54.020 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12bf141640 5 monclient: authenticate success, global_id 4322 2024-01-21T04:33:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12bf141640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.014+0000 7f12bf141640 4 set_mon_vals no callback set 2024-01-21T04:33:54.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.015+0000 7f12bf141640 10 monclient: shutdown 2024-01-21T04:33:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.015+0000 7f12bf141640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.015+0000 7f12bf141640 1 -- 172.21.15.69:0/1833220035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12b8142b90 msgr2=0x7f12b8141f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.015+0000 7f12bf141640 1 --2- 172.21.15.69:0/1833220035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12b8142b90 0x7f12b8141f30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f12a40097d0 tx=0x7f12a402b8a0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.015+0000 7f12bf141640 1 -- 172.21.15.69:0/1833220035 shutdown_connections 2024-01-21T04:33:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.015+0000 7f12bf141640 1 --2- 172.21.15.69:0/1833220035 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12b8142b90 0x7f12b8141f30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.015+0000 7f12bf141640 1 -- 172.21.15.69:0/1833220035 >> 172.21.15.69:0/1833220035 conn(0x7f12b80b23c0 msgr2=0x7f12b80b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.015+0000 7f12bf141640 1 -- 172.21.15.69:0/1833220035 shutdown_connections 2024-01-21T04:33:54.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.015+0000 7f12bf141640 1 -- 172.21.15.69:0/1833220035 wait complete. 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.016+0000 7f12bf141640 10 monclient: build_initial_monmap 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.016+0000 7f12bf141640 10 monclient: monmap: 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.016+0000 7f12bf141640 1 librados: starting msgr at 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.016+0000 7f12bf141640 1 librados: starting objecter 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.016+0000 7f12bf141640 1 Processor -- start 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.016+0000 7f12bf141640 1 -- start start 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.016+0000 7f12bf141640 1 librados: setting wanted keys 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.016+0000 7f12bf141640 1 librados: calling monclient init 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.016+0000 7f12bf141640 10 monclient: init 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12b80b12d0 0x7f12b8144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 10 monclient(hunting): picked mon.noname-a con 0x7f12b80b12d0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12b80b16b0 0x7f12b8136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 10 monclient(hunting): picked mon.noname-b con 0x7f12b80b16b0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12b8142b90 0x7f12b80adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 10 monclient(hunting): picked mon.noname-c con 0x7f12b8142b90 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:54.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80ae7f0 con 0x7f12b80b12d0 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b8144ca0 con 0x7f12b80b16b0 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80af1b0 con 0x7f12b8142b90 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.017+0000 7f12bf141640 10 monclient(hunting): authenticate will time out at 1392.358847s 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.018+0000 7f12b7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12b80b16b0 0x7f12b8136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.018+0000 7f12b7fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12b80b16b0 0x7f12b8136ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:60034/0 (socket says 172.21.15.69:60034) 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.018+0000 7f12b7fff640 1 -- 172.21.15.69:0/3047250838 learned_addr learned my addr 172.21.15.69:0/3047250838 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.018+0000 7f12b7fff640 10 monclient(hunting): get_auth_request con 0x7f12b80b16b0 auth_method 0 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.018+0000 7f12b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.018+0000 7f12b7fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:54.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.018+0000 7f12b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.018+0000 7f12bcedd640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12b80b12d0 0x7f12b8144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.019+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.019+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.019+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.019+0000 7f12bcedd640 10 monclient(hunting): get_auth_request con 0x7f12b80b12d0 auth_method 0 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.019+0000 7f12bcedd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.019+0000 7f12bcedd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.019+0000 7f12bcedd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.019+0000 7f12bcedd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.019+0000 7f12bcedd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.019+0000 7f12bcedd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.019+0000 7f12bd6de640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12b8142b90 0x7f12b80adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.020+0000 7f12bd6de640 10 monclient(hunting): get_auth_request con 0x7f12b8142b90 auth_method 0 2024-01-21T04:33:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.020+0000 7f12bd6de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.020+0000 7f12bd6de640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.020+0000 7f12bd6de640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12bd6de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12bd6de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12bd6de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12b7fff640 10 monclient(hunting): handle_auth_done global_id 4390 payload 824 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12b7fff640 1 -- 172.21.15.69:0/3047250838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12b8142b90 msgr2=0x7f12b80adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12b7fff640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12b8142b90 0x7f12b80adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12b7fff640 1 -- 172.21.15.69:0/3047250838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12b80b12d0 msgr2=0x7f12b8144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12b7fff640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12b80b12d0 0x7f12b8144760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12b7fff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12b7fff640 1 monclient: found mon.noname-b 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12b7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:54.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.021+0000 7f12b7fff640 1 -- 172.21.15.69:0/3047250838 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a4009400 con 0x7f12b80b16b0 2024-01-21T04:33:54.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.024+0000 7f12b7fff640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12b80b16b0 0x7f12b8136ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f12a40097d0 tx=0x7f12a40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:54.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.024+0000 7f12b57fa640 10 client.?.objecter ms_handle_connect 0x7f12b80b16b0 2024-01-21T04:33:54.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.024+0000 7f12b57fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.029+0000 7f12b57fa640 1 -- 172.21.15.69:0/3047250838 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a403ac60 con 0x7f12b80b16b0 2024-01-21T04:33:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:54.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 10 monclient: dump: 2024-01-21T04:33:54.032 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:54.032 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:54.032 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.032 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.032 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:54.032 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:54.033 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 10 monclient: _finish_auth 0 2024-01-21T04:33:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 10 monclient: _check_auth_tickets 2024-01-21T04:33:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:54.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 1 -- 172.21.15.69:0/3047250838 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a403ade0 con 0x7f12b80b16b0 2024-01-21T04:33:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 1 -- 172.21.15.69:0/3047250838 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4044630 con 0x7f12b80b16b0 2024-01-21T04:33:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b57fa640 10 monclient: dump: 2024-01-21T04:33:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:54.034 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:54.035 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12bf141640 5 monclient: authenticate success, global_id 4390 2024-01-21T04:33:54.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12bf141640 10 monclient: _renew_subs 2024-01-21T04:33:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12bf141640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12bf141640 1 -- 172.21.15.69:0/3047250838 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b80af490 con 0x7f12b80b16b0 2024-01-21T04:33:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12b67fc640 4 set_mon_vals no callback set 2024-01-21T04:33:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12bf141640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12bf141640 10 monclient: _renew_subs 2024-01-21T04:33:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12bf141640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12bf141640 1 -- 172.21.15.69:0/3047250838 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b80ad3f0 con 0x7f12b80b16b0 2024-01-21T04:33:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.030+0000 7f12bf141640 1 librados: init done 2024-01-21T04:33:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.031+0000 7f12bf141640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:54.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.031+0000 7f12bf141640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:54.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.031+0000 7f12bf141640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:54.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.031+0000 7f12bf141640 1 -- 172.21.15.69:0/3047250838 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b80ae930 con 0x7f12b80b16b0 2024-01-21T04:33:54.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.048+0000 7f12b57fa640 1 -- 172.21.15.69:0/3047250838 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f12a404d450 con 0x7f12b80b16b0 2024-01-21T04:33:54.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.048+0000 7f12b57fa640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f129c06c8b0 0x7f129c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:54.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.049+0000 7f12bcedd640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f129c06c8b0 0x7f129c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:54.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.049+0000 7f12bcedd640 10 monclient: get_auth_request con 0x7f129c06c8b0 auth_method 0 2024-01-21T04:33:54.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.049+0000 7f12bcedd640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f129c06c8b0 0x7f129c06ed70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0010e0 tx=0x7f12ac007be0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:54.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.049+0000 7f12b57fa640 10 client.4390.objecter ms_handle_connect 0x7f129c06c8b0 2024-01-21T04:33:54.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.054+0000 7f12b57fa640 1 -- 172.21.15.69:0/3047250838 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f12a402e070 con 0x7f12b80b16b0 2024-01-21T04:33:54.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.054+0000 7f12b57fa640 10 client.4390.objecter ms_dispatch 0x7f12b8000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:54.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.054+0000 7f12b57fa640 3 client.4390.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:54.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.054+0000 7f12b57fa640 3 client.4390.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:54.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.054+0000 7f12b57fa640 20 client.4390.objecter dump_active .. 0 homeless 2024-01-21T04:33:54.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.296+0000 7f12b57fa640 1 -- 172.21.15.69:0/3047250838 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f12a4076aa0 con 0x7f12b80b16b0 2024-01-21T04:33:54.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.296+0000 7f12b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:54.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.296+0000 7f12b57fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:54.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.487+0000 7f12bf141640 1 -- 172.21.15.69:0/3047250838 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f12b80ae930 con 0x7f129c06c8b0 2024-01-21T04:33:54.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.490+0000 7f12b57fa640 1 -- 172.21.15.69:0/3047250838 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47995 (secure 0 0 0) 0x7f12b80ae930 con 0x7f129c06c8b0 2024-01-21T04:33:54.491 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:33:54.492 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-01-21T04:33:54.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 10 librados: watch_flush enter 2024-01-21T04:33:54.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 10 librados: watch_flush exit 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 20 client.4390.objecter shutdown clearing up homeless session... 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 10 client.4390.objecter successfully canceled tick 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 -- 172.21.15.69:0/3047250838 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f129c06c8b0 msgr2=0x7f129c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f129c06c8b0 0x7f129c06ed70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0010e0 tx=0x7f12ac007be0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 10 monclient: shutdown 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 -- 172.21.15.69:0/3047250838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12b80b16b0 msgr2=0x7f12b8136ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12b80b16b0 0x7f12b8136ed0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f12a40097d0 tx=0x7f12a40046c0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 -- 172.21.15.69:0/3047250838 shutdown_connections 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f129c06c8b0 0x7f129c06ed70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f12b8142b90 0x7f12b80adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12b80b16b0 0x7f12b8136ed0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 --2- 172.21.15.69:0/3047250838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12b80b12d0 0x7f12b8144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 -- 172.21.15.69:0/3047250838 >> 172.21.15.69:0/3047250838 conn(0x7f12b80b23c0 msgr2=0x7f12b80b47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:54.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 -- 172.21.15.69:0/3047250838 shutdown_connections 2024-01-21T04:33:54.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 -- 172.21.15.69:0/3047250838 wait complete. 2024-01-21T04:33:54.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.492+0000 7f12bf141640 1 librados: shutdown 2024-01-21T04:33:54.503 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-01-21T04:33:53.412187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218856,"kb_used_data":2664,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755864,"statfs":{"total":773094113280,"available":772870004736,"internally_reserved":0,"allocated":2727936,"data_stored":1486008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":105,"apply_latency_ms":105,"commit_latency_ns":105000000,"apply_latency_ns":105000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001224"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.502765+0000","last_change":"2024-01-21T04:33:48.051441+0000","last_active":"2024-01-21T04:33:50.502765+0000","last_peered":"2024-01-21T04:33:50.502765+0000","last_clean":"2024-01-21T04:33:50.502765+0000","last_became_active":"2024-01-21T04:33:48.046404+0000","last_became_peered":"2024-01-21T04:33:48.046404+0000","last_unstale":"2024-01-21T04:33:50.502765+0000","last_undegraded":"2024-01-21T04:33:50.502765+0000","last_fullsized":"2024-01-21T04:33:50.502765+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:19:24.446344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-01-21T04:33:48.406521+0000","last_change":"2024-01-21T04:33:48.112047+0000","last_active":"2024-01-21T04:33:48.406521+0000","last_peered":"2024-01-21T04:33:48.406521+0000","last_clean":"2024-01-21T04:33:48.406521+0000","last_became_active":"2024-01-21T04:33:48.080898+0000","last_became_peered":"2024-01-21T04:33:48.080898+0000","last_unstale":"2024-01-21T04:33:48.406521+0000","last_undegraded":"2024-01-21T04:33:48.406521+0000","last_fullsized":"2024-01-21T04:33:48.406521+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:06:10.794664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.490433+0000","last_change":"2024-01-21T04:33:47.840628+0000","last_active":"2024-01-21T04:33:50.490433+0000","last_peered":"2024-01-21T04:33:50.490433+0000","last_clean":"2024-01-21T04:33:50.490433+0000","last_became_active":"2024-01-21T04:33:47.835639+0000","last_became_peered":"2024-01-21T04:33:47.835639+0000","last_unstale":"2024-01-21T04:33:50.490433+0000","last_undegraded":"2024-01-21T04:33:50.490433+0000","last_fullsized":"2024-01-21T04:33:50.490433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:55:33.768752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.309406+0000","last_change":"2024-01-21T04:33:48.068776+0000","last_active":"2024-01-21T04:33:50.309406+0000","last_peered":"2024-01-21T04:33:50.309406+0000","last_clean":"2024-01-21T04:33:50.309406+0000","last_became_active":"2024-01-21T04:33:48.032152+0000","last_became_peered":"2024-01-21T04:33:48.032152+0000","last_unstale":"2024-01-21T04:33:50.309406+0000","last_undegraded":"2024-01-21T04:33:50.309406+0000","last_fullsized":"2024-01-21T04:33:50.309406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:12:43.256934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.403490+0000","last_change":"2024-01-21T04:33:48.020958+0000","last_active":"2024-01-21T04:33:50.403490+0000","last_peered":"2024-01-21T04:33:50.403490+0000","last_clean":"2024-01-21T04:33:50.403490+0000","last_became_active":"2024-01-21T04:33:48.015845+0000","last_became_peered":"2024-01-21T04:33:48.015845+0000","last_unstale":"2024-01-21T04:33:50.403490+0000","last_undegraded":"2024-01-21T04:33:50.403490+0000","last_fullsized":"2024-01-21T04:33:50.403490+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:42:22.571268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-21T04:33:48.753025+0000","last_change":"2024-01-21T04:33:48.697481+0000","last_active":"2024-01-21T04:33:48.753025+0000","last_peered":"2024-01-21T04:33:48.753025+0000","last_clean":"2024-01-21T04:33:48.753025+0000","last_became_active":"2024-01-21T04:33:48.670864+0000","last_became_peered":"2024-01-21T04:33:48.670864+0000","last_unstale":"2024-01-21T04:33:48.753025+0000","last_undegraded":"2024-01-21T04:33:48.753025+0000","last_fullsized":"2024-01-21T04:33:48.753025+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:26:40.444643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.382508+0000","last_change":"2024-01-21T04:33:47.762316+0000","last_active":"2024-01-21T04:33:50.382508+0000","last_peered":"2024-01-21T04:33:50.382508+0000","last_clean":"2024-01-21T04:33:50.382508+0000","last_became_active":"2024-01-21T04:33:47.734733+0000","last_became_peered":"2024-01-21T04:33:47.734733+0000","last_unstale":"2024-01-21T04:33:50.382508+0000","last_undegraded":"2024-01-21T04:33:50.382508+0000","last_fullsized":"2024-01-21T04:33:50.382508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:16:54.133000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.399589+0000","last_change":"2024-01-21T04:33:48.369523+0000","last_active":"2024-01-21T04:33:50.399589+0000","last_peered":"2024-01-21T04:33:50.399589+0000","last_clean":"2024-01-21T04:33:50.399589+0000","last_became_active":"2024-01-21T04:33:48.364387+0000","last_became_peered":"2024-01-21T04:33:48.364387+0000","last_unstale":"2024-01-21T04:33:50.399589+0000","last_undegraded":"2024-01-21T04:33:50.399589+0000","last_fullsized":"2024-01-21T04:33:50.399589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:06:07.142172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":120,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-01-21T04:33:47.984543+0000","last_change":"2024-01-21T04:33:33.164364+0000","last_active":"2024-01-21T04:33:47.984543+0000","last_peered":"2024-01-21T04:33:47.984543+0000","last_clean":"2024-01-21T04:33:47.984543+0000","last_became_active":"2024-01-21T04:33:33.133732+0000","last_became_peered":"2024-01-21T04:33:33.133732+0000","last_unstale":"2024-01-21T04:33:47.984543+0000","last_undegraded":"2024-01-21T04:33:47.984543+0000","last_fullsized":"2024-01-21T04:33:47.984543+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:29.982915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:29.982915+0000","last_clean_scrub_stamp":"2024-01-21T04:33:29.982915+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:02:00.259080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":21,"seq":90194313226,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":84854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":89,"apply_latency_ms":89,"commit_latency_ns":89000000,"apply_latency_ns":89000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.641}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.381}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":194.863}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.152000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.370999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.768000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.494}]}]},{"osd":6,"up_from":20,"seq":85899345929,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":90602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8449999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.715999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.895}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.288}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.738}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.673999999999999}]}]},{"osd":4,"up_from":20,"seq":85899345928,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":84854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.937999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.154}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":146.893}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.692}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.013999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.801}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.147}]}]},{"osd":7,"up_from":15,"seq":64424509448,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":84854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.327999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4820000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.348000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.117999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5510000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.206}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.297000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":84854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.602}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.515999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.387}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.976000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":40.908000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":43.003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.263000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":84854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.598000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6870000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":36.039000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.524000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5399999999999991}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.324}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":283.98500000000001}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":482694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3650000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.574}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.975}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.016999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.776}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.561}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.573}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":651264,"data_stored":488442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9459999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8409999999999993}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.055999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.044}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.21}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.542999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.428999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T04:33:54.505 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T04:33:54.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.602+0000 7fc422dd0640 10 monclient: get_monmap_and_config 2024-01-21T04:33:54.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.602+0000 7fc422dd0640 10 monclient: build_initial_monmap 2024-01-21T04:33:54.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.602+0000 7fc422dd0640 10 monclient: monmap: 2024-01-21T04:33:54.603 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:54.603 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:54.604 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:54.603861+0000 2024-01-21T04:33:54.604 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:54.603861+0000 2024-01-21T04:33:54.604 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:54.604 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.604 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:54.604 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:54.604 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:54.604 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:54.604 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:54.604 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.603+0000 7fc422dd0640 1 Processor -- start 2024-01-21T04:33:54.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.603+0000 7fc422dd0640 1 -- start start 2024-01-21T04:33:54.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.603+0000 7fc422dd0640 10 monclient: init 2024-01-21T04:33:54.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc41c066620 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc41c0ac8f0 addr v1:172.21.15.69:6789/0 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc41c145af0 0x7fc41c14d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 10 monclient(hunting): picked mon.noname-c con 0x7fc41c145af0 addr v2:172.21.15.144:3300/0 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c138cb0 con 0x7fc41c0ac8f0 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c0ad2a0 con 0x7fc41c066620 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41c06f290 con 0x7fc41c145af0 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc422dd0640 10 monclient(hunting): authenticate will time out at 1392.945538s 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.604+0000 7fc42136d640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fc41c145af0 0x7fc41c14d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.605+0000 7fc41bfff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc41c0ac8f0 0x7fc41c1453e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55620/0 (socket says 172.21.15.69:55620) 2024-01-21T04:33:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.605+0000 7fc41bfff640 1 -- 172.21.15.69:0/3186560527 learned_addr learned my addr 172.21.15.69:0/3186560527 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:54.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.605+0000 7fc42136d640 10 monclient(hunting): get_auth_request con 0x7fc41c145af0 auth_method 0 2024-01-21T04:33:54.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.605+0000 7fc42136d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:54.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.605+0000 7fc42136d640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:54.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.605+0000 7fc42136d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:54.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.606+0000 7fc42136d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:54.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.606+0000 7fc42136d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:54.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.606+0000 7fc42136d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.607+0000 7fc42136d640 10 monclient(hunting): handle_auth_done global_id 4393 payload 274 2024-01-21T04:33:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.607+0000 7fc42136d640 1 -- 172.21.15.69:0/3186560527 >> v1:172.21.15.144:6789/0 conn(0x7fc41c066620 legacy=0x7fc41c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.608+0000 7fc42136d640 1 -- 172.21.15.69:0/3186560527 >> v1:172.21.15.69:6789/0 conn(0x7fc41c0ac8f0 legacy=0x7fc41c1453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.608+0000 7fc42136d640 10 monclient: _finish_hunting 0 2024-01-21T04:33:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.608+0000 7fc42136d640 1 monclient: found mon.noname-c 2024-01-21T04:33:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.608+0000 7fc42136d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.608+0000 7fc42136d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T04:33:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.608+0000 7fc42136d640 1 -- 172.21.15.69:0/3186560527 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c06f470 con 0x7fc41c145af0 2024-01-21T04:33:54.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.613+0000 7fc42136d640 1 --2- 172.21.15.69:0/3186560527 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc41c145af0 0x7fc41c14d780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0097a0 tx=0x7fc40c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cdf3a0533390c5d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:54.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 1 -- 172.21.15.69:0/3186560527 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c031830 con 0x7fc41c145af0 2024-01-21T04:33:54.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:54.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T04:33:54.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 10 monclient: dump: 2024-01-21T04:33:54.619 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:54.619 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:54.619 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 10 monclient: _finish_auth 0 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 10 monclient: _check_auth_tickets 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 1 -- 172.21.15.69:0/3186560527 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40c031990 con 0x7fc41c145af0 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 5 monclient: authenticate success, global_id 4393 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:54.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 1 -- 172.21.15.69:0/3186560527 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c031d20 con 0x7fc41c145af0 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 4 set_mon_vals no callback set 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc41b7fe640 10 monclient: dump: 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:54.621 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 10 monclient: shutdown 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 1 -- 172.21.15.69:0/3186560527 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc41c145af0 msgr2=0x7fc41c14d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 1 --2- 172.21.15.69:0/3186560527 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc41c145af0 0x7fc41c14d780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0097a0 tx=0x7fc40c02b8d0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 1 -- 172.21.15.69:0/3186560527 shutdown_connections 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 1 --2- 172.21.15.69:0/3186560527 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc41c145af0 0x7fc41c14d780 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 1 -- 172.21.15.69:0/3186560527 >> 172.21.15.69:0/3186560527 conn(0x7fc41c0b4190 msgr2=0x7fc41c0b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 1 -- 172.21.15.69:0/3186560527 shutdown_connections 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.617+0000 7fc422dd0640 1 -- 172.21.15.69:0/3186560527 wait complete. 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.618+0000 7fc422dd0640 10 monclient: build_initial_monmap 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.618+0000 7fc422dd0640 10 monclient: monmap: 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.618+0000 7fc422dd0640 1 librados: starting msgr at 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.618+0000 7fc422dd0640 1 librados: starting objecter 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.618+0000 7fc422dd0640 1 Processor -- start 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.618+0000 7fc422dd0640 1 -- start start 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.618+0000 7fc422dd0640 1 librados: setting wanted keys 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.618+0000 7fc422dd0640 1 librados: calling monclient init 2024-01-21T04:33:54.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.618+0000 7fc422dd0640 10 monclient: init 2024-01-21T04:33:54.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:54.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T04:33:54.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc41c066620 0x7fc41c13d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:54.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 10 monclient(hunting): picked mon.noname-b con 0x7fc41c066620 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:54.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c145af0 0x7fc41c13b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:54.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 10 monclient(hunting): picked mon.noname-a con 0x7fc41c145af0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:54.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41c13baa0 0x7fc41c1da920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 10 monclient(hunting): picked mon.noname-c con 0x7fc41c13baa0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c1d4ce0 con 0x7fc41c145af0 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c1411f0 con 0x7fc41c066620 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c13f2e0 con 0x7fc41c13baa0 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.619+0000 7fc422dd0640 10 monclient(hunting): authenticate will time out at 1392.960863s 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c145af0 0x7fc41c13b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc42136d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41c13baa0 0x7fc41c1da920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc420b6c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc41c066620 0x7fc41c13d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c145af0 0x7fc41c13b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59658/0 (socket says 172.21.15.69:59658) 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc41bfff640 1 -- 172.21.15.69:0/829253675 learned_addr learned my addr 172.21.15.69:0/829253675 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:54.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc41bfff640 10 monclient(hunting): get_auth_request con 0x7fc41c145af0 auth_method 0 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc41bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc41bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc41bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc42136d640 10 monclient(hunting): get_auth_request con 0x7fc41c13baa0 auth_method 0 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc42136d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc42136d640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc42136d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc420b6c640 10 monclient(hunting): get_auth_request con 0x7fc41c066620 auth_method 0 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc420b6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc420b6c640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.620+0000 7fc420b6c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.621+0000 7fc41bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.621+0000 7fc41bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:54.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.621+0000 7fc41bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.621+0000 7fc420b6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.621+0000 7fc420b6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.621+0000 7fc420b6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.621+0000 7fc42136d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.621+0000 7fc42136d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.621+0000 7fc42136d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.623+0000 7fc420b6c640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.623+0000 7fc420b6c640 1 -- 172.21.15.69:0/829253675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41c13baa0 msgr2=0x7fc41c1da920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.623+0000 7fc420b6c640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41c13baa0 0x7fc41c1da920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.623+0000 7fc420b6c640 1 -- 172.21.15.69:0/829253675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c145af0 msgr2=0x7fc41c13b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.623+0000 7fc420b6c640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c145af0 0x7fc41c13b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.623+0000 7fc420b6c640 10 monclient: _finish_hunting 0 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.623+0000 7fc420b6c640 1 monclient: found mon.noname-b 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.623+0000 7fc420b6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:54.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.623+0000 7fc420b6c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:54.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.623+0000 7fc420b6c640 1 -- 172.21.15.69:0/829253675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c009400 con 0x7fc41c066620 2024-01-21T04:33:54.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.625+0000 7fc420b6c640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc41c066620 0x7fc41c13d4f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc41400c770 tx=0x7fc41400cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:54.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.625+0000 7fc4197fa640 10 client.?.objecter ms_handle_connect 0x7fc41c066620 2024-01-21T04:33:54.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.625+0000 7fc4197fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 1 -- 172.21.15.69:0/829253675 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414003b70 con 0x7fc41c066620 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 10 monclient: dump: 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:54.631 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 10 monclient: _finish_auth 0 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 10 monclient: _check_auth_tickets 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 1 -- 172.21.15.69:0/829253675 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc414003cd0 con 0x7fc41c066620 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 1 -- 172.21.15.69:0/829253675 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41400aa30 con 0x7fc41c066620 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc4197fa640 10 monclient: dump: 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.632 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc422dd0640 5 monclient: authenticate success, global_id 4396 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc41affd640 4 set_mon_vals no callback set 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc422dd0640 10 monclient: _renew_subs 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc422dd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc422dd0640 1 -- 172.21.15.69:0/829253675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41c13df60 con 0x7fc41c066620 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc422dd0640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc422dd0640 10 monclient: _renew_subs 2024-01-21T04:33:54.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc422dd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:54.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc422dd0640 1 -- 172.21.15.69:0/829253675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c13cf60 con 0x7fc41c066620 2024-01-21T04:33:54.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.629+0000 7fc422dd0640 1 librados: init done 2024-01-21T04:33:54.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.630+0000 7fc422dd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:54.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.630+0000 7fc422dd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:54.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.630+0000 7fc422dd0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:54.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.630+0000 7fc422dd0640 1 -- 172.21.15.69:0/829253675 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41c13e180 con 0x7fc41c066620 2024-01-21T04:33:54.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.647+0000 7fc4197fa640 1 -- 172.21.15.69:0/829253675 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc41400ab90 con 0x7fc41c066620 2024-01-21T04:33:54.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.647+0000 7fc4197fa640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc3fc06c8e0 0x7fc3fc06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:54.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.648+0000 7fc41bfff640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc3fc06c8e0 0x7fc3fc06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:54.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.648+0000 7fc41bfff640 10 monclient: get_auth_request con 0x7fc3fc06c8e0 auth_method 0 2024-01-21T04:33:54.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.648+0000 7fc41bfff640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc3fc06c8e0 0x7fc3fc06eda0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0b2dc0 tx=0x7fc404007ee0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:54.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.648+0000 7fc4197fa640 10 client.4396.objecter ms_handle_connect 0x7fc3fc06c8e0 2024-01-21T04:33:54.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.650+0000 7fc4197fa640 1 -- 172.21.15.69:0/829253675 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc41400ae40 con 0x7fc41c066620 2024-01-21T04:33:54.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.650+0000 7fc4197fa640 10 client.4396.objecter ms_dispatch 0x7fc41c000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:54.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.650+0000 7fc4197fa640 3 client.4396.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:54.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.650+0000 7fc4197fa640 3 client.4396.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:54.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.650+0000 7fc4197fa640 20 client.4396.objecter dump_active .. 0 homeless 2024-01-21T04:33:54.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.886+0000 7fc4197fa640 1 -- 172.21.15.69:0/829253675 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fc414054d10 con 0x7fc41c066620 2024-01-21T04:33:54.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.886+0000 7fc4197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:54.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:54.886+0000 7fc4197fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:55.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.057+0000 7fc422dd0640 1 -- 172.21.15.69:0/829253675 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc41c13f450 con 0x7fc3fc06c8e0 2024-01-21T04:33:55.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.060+0000 7fc4197fa640 1 -- 172.21.15.69:0/829253675 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47995 (secure 0 0 0) 0x7fc41c13f450 con 0x7fc3fc06c8e0 2024-01-21T04:33:55.067 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:33:55.067 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-01-21T04:33:55.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.061+0000 7fc422dd0640 10 librados: watch_flush enter 2024-01-21T04:33:55.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.061+0000 7fc422dd0640 10 librados: watch_flush exit 2024-01-21T04:33:55.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.061+0000 7fc422dd0640 20 client.4396.objecter shutdown clearing up homeless session... 2024-01-21T04:33:55.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.061+0000 7fc422dd0640 10 client.4396.objecter successfully canceled tick 2024-01-21T04:33:55.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.061+0000 7fc422dd0640 1 -- 172.21.15.69:0/829253675 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc3fc06c8e0 msgr2=0x7fc3fc06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:55.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.061+0000 7fc422dd0640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc3fc06c8e0 0x7fc3fc06eda0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0b2dc0 tx=0x7fc404007ee0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.061+0000 7fc422dd0640 10 monclient: shutdown 2024-01-21T04:33:55.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.061+0000 7fc422dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:55.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.061+0000 7fc422dd0640 1 -- 172.21.15.69:0/829253675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc41c066620 msgr2=0x7fc41c13d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:55.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.061+0000 7fc422dd0640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc41c066620 0x7fc41c13d4f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc41400c770 tx=0x7fc41400cc30 comp rx=0 tx=0).stop 2024-01-21T04:33:55.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.062+0000 7fc422dd0640 1 -- 172.21.15.69:0/829253675 shutdown_connections 2024-01-21T04:33:55.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.062+0000 7fc422dd0640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc3fc06c8e0 0x7fc3fc06eda0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.062+0000 7fc422dd0640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc41c13baa0 0x7fc41c1da920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.062+0000 7fc422dd0640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c145af0 0x7fc41c13b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.062+0000 7fc422dd0640 1 --2- 172.21.15.69:0/829253675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc41c066620 0x7fc41c13d4f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.062+0000 7fc422dd0640 1 -- 172.21.15.69:0/829253675 >> 172.21.15.69:0/829253675 conn(0x7fc41c0b4190 msgr2=0x7fc41c1431e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:55.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.062+0000 7fc422dd0640 1 -- 172.21.15.69:0/829253675 shutdown_connections 2024-01-21T04:33:55.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.062+0000 7fc422dd0640 1 -- 172.21.15.69:0/829253675 wait complete. 2024-01-21T04:33:55.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.062+0000 7fc422dd0640 1 librados: shutdown 2024-01-21T04:33:55.073 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-01-21T04:33:53.412187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218856,"kb_used_data":2664,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755864,"statfs":{"total":773094113280,"available":772870004736,"internally_reserved":0,"allocated":2727936,"data_stored":1486008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":105,"apply_latency_ms":105,"commit_latency_ns":105000000,"apply_latency_ns":105000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001224"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.502765+0000","last_change":"2024-01-21T04:33:48.051441+0000","last_active":"2024-01-21T04:33:50.502765+0000","last_peered":"2024-01-21T04:33:50.502765+0000","last_clean":"2024-01-21T04:33:50.502765+0000","last_became_active":"2024-01-21T04:33:48.046404+0000","last_became_peered":"2024-01-21T04:33:48.046404+0000","last_unstale":"2024-01-21T04:33:50.502765+0000","last_undegraded":"2024-01-21T04:33:50.502765+0000","last_fullsized":"2024-01-21T04:33:50.502765+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:19:24.446344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-01-21T04:33:48.406521+0000","last_change":"2024-01-21T04:33:48.112047+0000","last_active":"2024-01-21T04:33:48.406521+0000","last_peered":"2024-01-21T04:33:48.406521+0000","last_clean":"2024-01-21T04:33:48.406521+0000","last_became_active":"2024-01-21T04:33:48.080898+0000","last_became_peered":"2024-01-21T04:33:48.080898+0000","last_unstale":"2024-01-21T04:33:48.406521+0000","last_undegraded":"2024-01-21T04:33:48.406521+0000","last_fullsized":"2024-01-21T04:33:48.406521+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:06:10.794664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.490433+0000","last_change":"2024-01-21T04:33:47.840628+0000","last_active":"2024-01-21T04:33:50.490433+0000","last_peered":"2024-01-21T04:33:50.490433+0000","last_clean":"2024-01-21T04:33:50.490433+0000","last_became_active":"2024-01-21T04:33:47.835639+0000","last_became_peered":"2024-01-21T04:33:47.835639+0000","last_unstale":"2024-01-21T04:33:50.490433+0000","last_undegraded":"2024-01-21T04:33:50.490433+0000","last_fullsized":"2024-01-21T04:33:50.490433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:55:33.768752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.309406+0000","last_change":"2024-01-21T04:33:48.068776+0000","last_active":"2024-01-21T04:33:50.309406+0000","last_peered":"2024-01-21T04:33:50.309406+0000","last_clean":"2024-01-21T04:33:50.309406+0000","last_became_active":"2024-01-21T04:33:48.032152+0000","last_became_peered":"2024-01-21T04:33:48.032152+0000","last_unstale":"2024-01-21T04:33:50.309406+0000","last_undegraded":"2024-01-21T04:33:50.309406+0000","last_fullsized":"2024-01-21T04:33:50.309406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:12:43.256934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.403490+0000","last_change":"2024-01-21T04:33:48.020958+0000","last_active":"2024-01-21T04:33:50.403490+0000","last_peered":"2024-01-21T04:33:50.403490+0000","last_clean":"2024-01-21T04:33:50.403490+0000","last_became_active":"2024-01-21T04:33:48.015845+0000","last_became_peered":"2024-01-21T04:33:48.015845+0000","last_unstale":"2024-01-21T04:33:50.403490+0000","last_undegraded":"2024-01-21T04:33:50.403490+0000","last_fullsized":"2024-01-21T04:33:50.403490+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:42:22.571268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-21T04:33:48.753025+0000","last_change":"2024-01-21T04:33:48.697481+0000","last_active":"2024-01-21T04:33:48.753025+0000","last_peered":"2024-01-21T04:33:48.753025+0000","last_clean":"2024-01-21T04:33:48.753025+0000","last_became_active":"2024-01-21T04:33:48.670864+0000","last_became_peered":"2024-01-21T04:33:48.670864+0000","last_unstale":"2024-01-21T04:33:48.753025+0000","last_undegraded":"2024-01-21T04:33:48.753025+0000","last_fullsized":"2024-01-21T04:33:48.753025+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:26:40.444643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.382508+0000","last_change":"2024-01-21T04:33:47.762316+0000","last_active":"2024-01-21T04:33:50.382508+0000","last_peered":"2024-01-21T04:33:50.382508+0000","last_clean":"2024-01-21T04:33:50.382508+0000","last_became_active":"2024-01-21T04:33:47.734733+0000","last_became_peered":"2024-01-21T04:33:47.734733+0000","last_unstale":"2024-01-21T04:33:50.382508+0000","last_undegraded":"2024-01-21T04:33:50.382508+0000","last_fullsized":"2024-01-21T04:33:50.382508+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:16:54.133000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-01-21T04:33:50.399589+0000","last_change":"2024-01-21T04:33:48.369523+0000","last_active":"2024-01-21T04:33:50.399589+0000","last_peered":"2024-01-21T04:33:50.399589+0000","last_clean":"2024-01-21T04:33:50.399589+0000","last_became_active":"2024-01-21T04:33:48.364387+0000","last_became_peered":"2024-01-21T04:33:48.364387+0000","last_unstale":"2024-01-21T04:33:50.399589+0000","last_undegraded":"2024-01-21T04:33:50.399589+0000","last_fullsized":"2024-01-21T04:33:50.399589+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:06:07.142172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":120,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-01-21T04:33:47.984543+0000","last_change":"2024-01-21T04:33:33.164364+0000","last_active":"2024-01-21T04:33:47.984543+0000","last_peered":"2024-01-21T04:33:47.984543+0000","last_clean":"2024-01-21T04:33:47.984543+0000","last_became_active":"2024-01-21T04:33:33.133732+0000","last_became_peered":"2024-01-21T04:33:33.133732+0000","last_unstale":"2024-01-21T04:33:47.984543+0000","last_undegraded":"2024-01-21T04:33:47.984543+0000","last_fullsized":"2024-01-21T04:33:47.984543+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:29.982915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:29.982915+0000","last_clean_scrub_stamp":"2024-01-21T04:33:29.982915+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:02:00.259080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":21,"seq":90194313226,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":84854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":89,"apply_latency_ms":89,"commit_latency_ns":89000000,"apply_latency_ns":89000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.641}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.381}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":194.863}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.152000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.370999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.768000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.494}]}]},{"osd":6,"up_from":20,"seq":85899345929,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":90602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8449999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.715999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.895}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.288}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.738}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.673999999999999}]}]},{"osd":4,"up_from":20,"seq":85899345928,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":84854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.937999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.154}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":146.893}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.692}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.013999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.801}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.147}]}]},{"osd":7,"up_from":15,"seq":64424509448,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":84854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.327999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4820000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.348000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.117999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5510000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.206}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.297000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":84854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.602}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.515999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.387}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.976000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":40.908000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":43.003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.263000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":84854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.598000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6870000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":36.039000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.524000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5399999999999991}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.324}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":283.98500000000001}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":482694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.3650000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.574}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.975}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.016999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.776}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.561}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.573}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":651264,"data_stored":488442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9459999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8409999999999993}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.055999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.044}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.21}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.542999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.428999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T04:33:55.074 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-21T04:33:55.074 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-21T04:33:55.074 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-21T04:33:55.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-21T04:33:55.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.172+0000 7fa7927d7640 10 monclient: get_monmap_and_config 2024-01-21T04:33:55.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.172+0000 7fa7927d7640 10 monclient: build_initial_monmap 2024-01-21T04:33:55.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.172+0000 7fa7927d7640 10 monclient: monmap: 2024-01-21T04:33:55.173 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:55.173 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:55.173 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:33:55.173192+0000 2024-01-21T04:33:55.173 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:33:55.173192+0000 2024-01-21T04:33:55.173 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:55.174 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:55.174 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T04:33:55.174 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T04:33:55.174 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T04:33:55.174 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T04:33:55.174 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T04:33:55.174 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.172+0000 7fa7927d7640 1 Processor -- start 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.172+0000 7fa7927d7640 1 -- start start 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.172+0000 7fa7927d7640 10 monclient: init 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa78c066620 0x7fa78c071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 10 monclient(hunting): picked mon.noname-a con 0x7fa78c066620 addr v2:172.21.15.69:3300/0 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78c0ac8f0 0x7fa78c1453e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 10 monclient(hunting): picked mon.noname-b con 0x7fa78c0ac8f0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa78c072330 addr v1:172.21.15.144:6789/0 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78c138cb0 con 0x7fa78c072330 2024-01-21T04:33:55.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa78c06f290 con 0x7fa78c066620 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c066a00 con 0x7fa78c0ac8f0 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.173+0000 7fa7927d7640 10 monclient(hunting): authenticate will time out at 1393.514761s 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa78bfff640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa78c066620 0x7fa78c071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78c0ac8f0 0x7fa78c1453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa790d74640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fa78c072330 0x7fa78c145920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:46006/0 (socket says 172.21.15.69:46006) 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa790d74640 1 -- 172.21.15.69:0/2969356951 learned_addr learned my addr 172.21.15.69:0/2969356951 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa78bfff640 10 monclient(hunting): get_auth_request con 0x7fa78c066620 auth_method 0 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa78bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa78bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa78bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa78b7fe640 10 monclient(hunting): get_auth_request con 0x7fa78c0ac8f0 auth_method 0 2024-01-21T04:33:55.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa78b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:55.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa78b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:55.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.174+0000 7fa78b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:55.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.175+0000 7fa78bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:55.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.175+0000 7fa78bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:55.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.175+0000 7fa78bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:55.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.175+0000 7fa78b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:55.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.175+0000 7fa78b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:55.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.175+0000 7fa78b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:55.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78affd640 1 -- 172.21.15.69:0/2969356951 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550241545 0 0) 0x7fa78c138cb0 con 0x7fa78c072330 2024-01-21T04:33:55.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78affd640 10 monclient(hunting): _negotiate 2024-01-21T04:33:55.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:55.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:55.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78affd640 10 monclient(hunting): my global_id is 4399 2024-01-21T04:33:55.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78affd640 1 -- 172.21.15.69:0/2969356951 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa774003520 con 0x7fa78c072330 2024-01-21T04:33:55.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78bfff640 10 monclient(hunting): handle_auth_done global_id 4383 payload 274 2024-01-21T04:33:55.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78bfff640 1 -- 172.21.15.69:0/2969356951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78c0ac8f0 msgr2=0x7fa78c1453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:33:55.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78bfff640 1 --2- 172.21.15.69:0/2969356951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78c0ac8f0 0x7fa78c1453e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78bfff640 1 -- 172.21.15.69:0/2969356951 >> v1:172.21.15.144:6789/0 conn(0x7fa78c072330 legacy=0x7fa78c145920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:55.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78bfff640 10 monclient: _finish_hunting 0 2024-01-21T04:33:55.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78bfff640 1 monclient: found mon.noname-a 2024-01-21T04:33:55.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:55.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:33:55.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.176+0000 7fa78bfff640 1 -- 172.21.15.69:0/2969356951 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c0ad2a0 con 0x7fa78c066620 2024-01-21T04:33:55.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.178+0000 7fa78bfff640 1 --2- 172.21.15.69:0/2969356951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78c066620 0x7fa78c071df0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa7800097d0 tx=0x7fa78002b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a78d6ed7d00971 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:55.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 1 -- 172.21.15.69:0/2969356951 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780031770 con 0x7fa78c066620 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 10 monclient: dump: 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:55.184 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 10 monclient: _finish_auth 0 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 10 monclient: _check_auth_tickets 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 1 -- 172.21.15.69:0/2969356951 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7800318d0 con 0x7fa78c066620 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 1 -- 172.21.15.69:0/2969356951 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780031c60 con 0x7fa78c066620 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa7927d7640 5 monclient: authenticate success, global_id 4383 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa7927d7640 10 monclient: get_monmap_and_config success 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa7927d7640 4 set_mon_vals no callback set 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa78affd640 10 monclient: dump: 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:55.185 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa7927d7640 10 monclient: shutdown 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa7927d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa7927d7640 1 -- 172.21.15.69:0/2969356951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78c066620 msgr2=0x7fa78c071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.182+0000 7fa7927d7640 1 --2- 172.21.15.69:0/2969356951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78c066620 0x7fa78c071df0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa7800097d0 tx=0x7fa78002b810 comp rx=0 tx=0).stop 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.183+0000 7fa7927d7640 1 -- 172.21.15.69:0/2969356951 shutdown_connections 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.183+0000 7fa7927d7640 1 --2- 172.21.15.69:0/2969356951 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78c0ac8f0 0x7fa78c1453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.183+0000 7fa7927d7640 1 --2- 172.21.15.69:0/2969356951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78c066620 0x7fa78c071df0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.183+0000 7fa7927d7640 1 -- 172.21.15.69:0/2969356951 >> 172.21.15.69:0/2969356951 conn(0x7fa78c0b4190 msgr2=0x7fa78c0b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:55.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.183+0000 7fa7927d7640 1 -- 172.21.15.69:0/2969356951 shutdown_connections 2024-01-21T04:33:55.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.183+0000 7fa7927d7640 1 -- 172.21.15.69:0/2969356951 wait complete. 2024-01-21T04:33:55.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.183+0000 7fa7927d7640 10 monclient: build_initial_monmap 2024-01-21T04:33:55.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.183+0000 7fa7927d7640 10 monclient: monmap: 2024-01-21T04:33:55.189 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 librados: starting msgr at 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 librados: starting objecter 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 Processor -- start 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 -- start start 2024-01-21T04:33:55.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 librados: setting wanted keys 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 librados: calling monclient init 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 10 monclient: init 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 10 monclient: _reopen_session rank -1 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78c066620 0x7fa78c13b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 10 monclient(hunting): picked mon.noname-a con 0x7fa78c066620 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa78c072330 0x7fa78c13c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 10 monclient(hunting): picked mon.noname-b con 0x7fa78c072330 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78c139960 0x7fa78c13d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 10 monclient(hunting): picked mon.noname-c con 0x7fa78c139960 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c139240 con 0x7fa78c066620 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c13d380 con 0x7fa78c072330 2024-01-21T04:33:55.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 10 monclient(hunting): start opening mon connection 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c13aff0 con 0x7fa78c139960 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 10 monclient(hunting): _renew_subs 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.184+0000 7fa7927d7640 10 monclient(hunting): authenticate will time out at 1393.526156s 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.185+0000 7fa78bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78c066620 0x7fa78c13b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.185+0000 7fa790d74640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78c139960 0x7fa78c13d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.185+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa78c072330 0x7fa78c13c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.185+0000 7fa78bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78c066620 0x7fa78c13b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59674/0 (socket says 172.21.15.69:59674) 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.185+0000 7fa78bfff640 1 -- 172.21.15.69:0/2493670568 learned_addr learned my addr 172.21.15.69:0/2493670568 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.185+0000 7fa78bfff640 10 monclient(hunting): get_auth_request con 0x7fa78c066620 auth_method 0 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.185+0000 7fa78bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.185+0000 7fa78bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.185+0000 7fa78bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa790d74640 10 monclient(hunting): get_auth_request con 0x7fa78c139960 auth_method 0 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa790d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:55.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa790d74640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa790d74640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa78b7fe640 10 monclient(hunting): get_auth_request con 0x7fa78c072330 auth_method 0 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa78b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa78b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa78b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa78bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa78bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa78bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa78b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa78b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa78b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa790d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:33:55.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa790d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:33:55.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.186+0000 7fa790d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:33:55.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.188+0000 7fa78b7fe640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2024-01-21T04:33:55.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.188+0000 7fa78b7fe640 1 -- 172.21.15.69:0/2493670568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78c139960 msgr2=0x7fa78c13d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:55.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.188+0000 7fa78b7fe640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78c139960 0x7fa78c13d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.188+0000 7fa78b7fe640 1 -- 172.21.15.69:0/2493670568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78c066620 msgr2=0x7fa78c13b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:55.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.188+0000 7fa78b7fe640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78c066620 0x7fa78c13b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.188+0000 7fa78b7fe640 10 monclient: _finish_hunting 0 2024-01-21T04:33:55.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.188+0000 7fa78b7fe640 1 monclient: found mon.noname-b 2024-01-21T04:33:55.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.188+0000 7fa78b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:33:55.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.188+0000 7fa78b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:33:55.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.188+0000 7fa78b7fe640 1 -- 172.21.15.69:0/2493670568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa780009400 con 0x7fa78c072330 2024-01-21T04:33:55.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.191+0000 7fa78b7fe640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa78c072330 0x7fa78c13c9d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00e8a0 tx=0x7fa77c00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:55.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.191+0000 7fa788ff9640 10 client.?.objecter ms_handle_connect 0x7fa78c072330 2024-01-21T04:33:55.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.191+0000 7fa788ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T04:33:55.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 1 -- 172.21.15.69:0/2493670568 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c009e60 con 0x7fa78c072330 2024-01-21T04:33:55.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:55.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:33:55.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 10 monclient: dump: 2024-01-21T04:33:55.196 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:55.196 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:55.196 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:55.196 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:55.196 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:55.196 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 10 monclient: _finish_auth 0 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 10 monclient: _check_auth_tickets 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 1 -- 172.21.15.69:0/2493670568 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77c00c940 con 0x7fa78c072330 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 1 -- 172.21.15.69:0/2493670568 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c00ccd0 con 0x7fa78c072330 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa7927d7640 5 monclient: authenticate success, global_id 4402 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa7927d7640 10 monclient: _renew_subs 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa7927d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa7927d7640 1 -- 172.21.15.69:0/2493670568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78c138e60 con 0x7fa78c072330 2024-01-21T04:33:55.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa788ff9640 10 monclient: dump: 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa78a7fc640 4 set_mon_vals no callback set 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa7927d7640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa7927d7640 10 monclient: _renew_subs 2024-01-21T04:33:55.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa7927d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:55.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa7927d7640 1 -- 172.21.15.69:0/2493670568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78c13bd10 con 0x7fa78c072330 2024-01-21T04:33:55.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.194+0000 7fa7927d7640 1 librados: init done 2024-01-21T04:33:55.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.195+0000 7fa7927d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:55.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.195+0000 7fa7927d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:55.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.195+0000 7fa7927d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:55.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.195+0000 7fa7927d7640 1 -- 172.21.15.69:0/2493670568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78c04e970 con 0x7fa78c072330 2024-01-21T04:33:55.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.217+0000 7fa788ff9640 1 -- 172.21.15.69:0/2493670568 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fa77c013470 con 0x7fa78c072330 2024-01-21T04:33:55.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.217+0000 7fa788ff9640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa77806c8b0 0x7fa77806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:33:55.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.218+0000 7fa78bfff640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa77806c8b0 0x7fa77806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:33:55.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.218+0000 7fa78bfff640 10 monclient: get_auth_request con 0x7fa77806c8b0 auth_method 0 2024-01-21T04:33:55.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.218+0000 7fa788ff9640 10 client.4402.objecter ms_handle_connect 0x7fa77806c8b0 2024-01-21T04:33:55.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.218+0000 7fa78bfff640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa77806c8b0 0x7fa77806ed70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa78002b850 tx=0x7fa78003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:33:55.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.444+0000 7fa788ff9640 1 -- 172.21.15.69:0/2493670568 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa77c0589a0 con 0x7fa78c072330 2024-01-21T04:33:55.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.444+0000 7fa788ff9640 10 client.4402.objecter ms_dispatch 0x7fa78c000b90 osd_map(26..26 src has 1..26) v4 2024-01-21T04:33:55.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.444+0000 7fa788ff9640 3 client.4402.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:33:55.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.444+0000 7fa788ff9640 3 client.4402.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:33:55.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.444+0000 7fa788ff9640 20 client.4402.objecter dump_active .. 0 homeless 2024-01-21T04:33:55.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.454+0000 7fa788ff9640 1 -- 172.21.15.69:0/2493670568 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7fa77c054ce0 con 0x7fa78c072330 2024-01-21T04:33:55.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.454+0000 7fa788ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T04:33:55.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.454+0000 7fa788ff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T04:33:55.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.673+0000 7fa7927d7640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-01-21T04:33:55.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.673+0000 7fa7927d7640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-01-21T04:33:55.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.673+0000 7fa7927d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T04:33:55.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.673+0000 7fa7927d7640 1 -- 172.21.15.69:0/2493670568 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa78c070270 con 0x7fa78c072330 2024-01-21T04:33:55.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.680+0000 7fa788ff9640 1 -- 172.21.15.69:0/2493670568 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa77c017080 con 0x7fa78c072330 2024-01-21T04:33:55.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.680+0000 7fa788ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-01-21T04:33:55.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.680+0000 7fa788ff9640 10 monclient: _finish_command 2 = system:0 2024-01-21T04:33:55.681 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T04:33:55.681 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-21T04:33:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.681+0000 7fa7927d7640 10 librados: watch_flush enter 2024-01-21T04:33:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.681+0000 7fa7927d7640 10 librados: watch_flush exit 2024-01-21T04:33:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.681+0000 7fa7927d7640 20 client.4402.objecter shutdown clearing up homeless session... 2024-01-21T04:33:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.681+0000 7fa7927d7640 10 client.4402.objecter successfully canceled tick 2024-01-21T04:33:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.681+0000 7fa7927d7640 1 -- 172.21.15.69:0/2493670568 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa77806c8b0 msgr2=0x7fa77806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa77806c8b0 0x7fa77806ed70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa78002b850 tx=0x7fa78003d040 comp rx=0 tx=0).stop 2024-01-21T04:33:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 10 monclient: shutdown 2024-01-21T04:33:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:33:55.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 -- 172.21.15.69:0/2493670568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa78c072330 msgr2=0x7fa78c13c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:33:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa78c072330 0x7fa78c13c9d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00e8a0 tx=0x7fa77c00ed60 comp rx=0 tx=0).stop 2024-01-21T04:33:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 -- 172.21.15.69:0/2493670568 shutdown_connections 2024-01-21T04:33:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fa77806c8b0 0x7fa77806ed70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fa78c139960 0x7fa78c13d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa78c072330 0x7fa78c13c9d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 --2- 172.21.15.69:0/2493670568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa78c066620 0x7fa78c13b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:33:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 -- 172.21.15.69:0/2493670568 >> 172.21.15.69:0/2493670568 conn(0x7fa78c0b4190 msgr2=0x7fa78c0b2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:33:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 -- 172.21.15.69:0/2493670568 shutdown_connections 2024-01-21T04:33:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 -- 172.21.15.69:0/2493670568 wait complete. 2024-01-21T04:33:55.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T04:33:55.682+0000 7fa7927d7640 1 librados: shutdown 2024-01-21T04:33:55.693 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-21T04:33:55.693 INFO:teuthology.run_tasks:Running task workunit... 2024-01-21T04:33:55.707 INFO:tasks.workunit:Pulling workunits from ref 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T04:33:55.708 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-01-21T04:33:55.708 DEBUG:teuthology.orchestra.run.smithi069:> stat -- /home/ubuntu/cephtest/mnt.0 2024-01-21T04:33:55.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T04:33:55.722 INFO:teuthology.orchestra.run.smithi069.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-01-21T04:33:55.723 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-01-21T04:33:55.776 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-01-21T04:33:55.777 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-01-21T04:33:55.830 INFO:tasks.workunit:timeout=6h 2024-01-21T04:33:55.830 INFO:tasks.workunit:cleanup=True 2024-01-21T04:33:55.831 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0edf41a622739843d6b978b179ff3227b476dd9d 2024-01-21T04:33:55.885 INFO:tasks.workunit.client.0.smithi069.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-01-21T04:34:01.501 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:29.408 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:01.505 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:29.412 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:01.548 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:01:29.423 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:01.551 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:01:29.426 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:01.559 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:29.466 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:01.559 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:29.466 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:01.564 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:29.471 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:01.589 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:01:29.465 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:01.610 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:29.517 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:26.628 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:54.535 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:26.635 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:54.542 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:26.658 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:54.565 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:26.662 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:01:54.569 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:26.682 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:01:54.557 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:26.688 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:01:54.563 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:26.704 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:01:54.579 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:26.719 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:54.622 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:26.742 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:01:54.649 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:42.413 INFO:tasks.workunit.client.0.smithi069.stderr:Note: switching to '0edf41a622739843d6b978b179ff3227b476dd9d'. 2024-01-21T04:34:42.413 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-01-21T04:34:42.413 INFO:tasks.workunit.client.0.smithi069.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-01-21T04:34:42.413 INFO:tasks.workunit.client.0.smithi069.stderr:changes and commit them, and you can discard any commits you make in this 2024-01-21T04:34:42.413 INFO:tasks.workunit.client.0.smithi069.stderr:state without impacting any branches by switching back to a branch. 2024-01-21T04:34:42.413 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-01-21T04:34:42.413 INFO:tasks.workunit.client.0.smithi069.stderr:If you want to create a new branch to retain commits you create, you may 2024-01-21T04:34:42.413 INFO:tasks.workunit.client.0.smithi069.stderr:do so (now or later) by using -c with the switch command. Example: 2024-01-21T04:34:42.413 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-01-21T04:34:42.414 INFO:tasks.workunit.client.0.smithi069.stderr: git switch -c 2024-01-21T04:34:42.414 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-01-21T04:34:42.414 INFO:tasks.workunit.client.0.smithi069.stderr:Or undo this operation with: 2024-01-21T04:34:42.414 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-01-21T04:34:42.414 INFO:tasks.workunit.client.0.smithi069.stderr: git switch - 2024-01-21T04:34:42.414 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-01-21T04:34:42.414 INFO:tasks.workunit.client.0.smithi069.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-01-21T04:34:42.414 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-01-21T04:34:42.414 INFO:tasks.workunit.client.0.smithi069.stderr:HEAD is now at 0edf41a6227 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2024-01-18-0823 2024-01-21T04:34:42.417 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-01-21T04:34:42.483 INFO:tasks.workunit.client.0.smithi069.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-01-21T04:34:42.485 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-21T04:34:42.485 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-01-21T04:34:43.131 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-01-21T04:34:43.195 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-01-21T04:34:43.235 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-01-21T04:34:43.237 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-21T04:34:43.237 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-01-21T04:34:43.275 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-01-21T04:34:43.278 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-01-21T04:34:43.278 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-01-21T04:34:43.333 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-01-21T04:34:43.334 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-01-21T04:34:43.334 DEBUG:teuthology.orchestra.run.smithi069:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0edf41a622739843d6b978b179ff3227b476dd9d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-01-21T04:34:43.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ parallel=1 2024-01-21T04:34:43.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' '' = --serial ']' 2024-01-21T04:34:43.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ crimson=0 2024-01-21T04:34:43.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' '' = --crimson ']' 2024-01-21T04:34:43.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ color= 2024-01-21T04:34:43.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -t 1 ']' 2024-01-21T04:34:43.392 INFO:tasks.workunit.client.0.smithi069.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-01-21T04:34:43.393 INFO:tasks.workunit.client.0.smithi069.stderr:+ GTEST_OUTPUT=xml:/home/ubuntu/cephtest/archive/gtest_xml_report 2024-01-21T04:34:43.393 INFO:tasks.workunit.client.0.smithi069.stderr:+ declare -A pids 2024-01-21T04:34:43.393 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.393 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.393 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_aio 2024-01-21T04:34:43.393 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_aio' 2024-01-21T04:34:43.393 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_aio 2024-01-21T04:34:43.393 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.394 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_aio 2024-01-21T04:34:43.394 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39781 2024-01-21T04:34:43.394 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_aio on pid 39781' 2024-01-21T04:34:43.394 INFO:tasks.workunit.client.0.smithi069.stdout:test api_aio on pid 39781 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39781 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_aio_pp 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_aio_pp' 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_aio_pp 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.397 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.398 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.399 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.399 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.399 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.399 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.399 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_aio_pp 2024-01-21T04:34:43.399 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39788 2024-01-21T04:34:43.399 INFO:tasks.workunit.client.0.smithi069.stdout:test api_aio_pp on pid 39788 2024-01-21T04:34:43.403 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_aio_pp on pid 39788' 2024-01-21T04:34:43.403 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39788 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_io 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_io' 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_io 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.404 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.405 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.406 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_io 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39797 2024-01-21T04:34:43.407 INFO:tasks.workunit.client.0.smithi069.stdout:test api_io on pid 39797 2024-01-21T04:34:43.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_io on pid 39797' 2024-01-21T04:34:43.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39797 2024-01-21T04:34:43.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-01-21T04:34:43.411 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_io_pp 2024-01-21T04:34:43.411 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_io_pp' 2024-01-21T04:34:43.411 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_io_pp 2024-01-21T04:34:43.411 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.412 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.413 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.414 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_io_pp 2024-01-21T04:34:43.415 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39810 2024-01-21T04:34:43.416 INFO:tasks.workunit.client.0.smithi069.stdout:test api_io_pp on pid 39810 2024-01-21T04:34:43.421 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_io_pp on pid 39810' 2024-01-21T04:34:43.421 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39810 2024-01-21T04:34:43.421 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.421 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.421 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_asio 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_asio' 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.422 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_asio 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.423 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.424 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.425 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.426 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_asio 2024-01-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39826 2024-01-21T04:34:43.427 INFO:tasks.workunit.client.0.smithi069.stdout:test api_asio on pid 39826 2024-01-21T04:34:43.434 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_asio on pid 39826' 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39826 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_list 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.435 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_list' 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.436 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.437 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_list 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.438 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39781/exe 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.439 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.440 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.441 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_list 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.442 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39842 2024-01-21T04:34:43.443 INFO:tasks.workunit.client.0.smithi069.stdout:test api_list on pid 39842 2024-01-21T04:34:43.451 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_list on pid 39842' 2024-01-21T04:34:43.451 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.451 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39842 2024-01-21T04:34:43.451 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.451 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.451 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.451 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.451 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-01-21T04:34:43.451 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_lock 2024-01-21T04:34:43.451 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_lock' 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_aio: /' 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.452 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_aio.log 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.453 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.454 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.455 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_lock 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.456 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.457 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39788/exe 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.458 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_lock 2024-01-21T04:34:43.459 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.460 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.460 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.460 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.460 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.460 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.460 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.460 INFO:tasks.workunit.client.0.smithi069.stdout:test api_lock on pid 39862 2024-01-21T04:34:43.468 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39862 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_lock on pid 39862' 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39862 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.469 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_lock_pp 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.470 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_lock_pp' 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.471 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_lock_pp 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.472 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39797/exe 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.473 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_aio_pp: /' 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.474 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.475 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.476 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.477 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_lock_pp 2024-01-21T04:34:43.478 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.478 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.478 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.478 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.478 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.478 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39886 2024-01-21T04:34:43.478 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_lock_pp on pid 39886' 2024-01-21T04:34:43.478 INFO:tasks.workunit.client.0.smithi069.stdout:test api_lock_pp on pid 39886 2024-01-21T04:34:43.486 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39886 2024-01-21T04:34:43.486 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.486 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.486 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-01-21T04:34:43.486 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.486 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.486 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.486 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.486 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.486 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_misc 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_misc' 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.487 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_io: /' 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_io.log 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.488 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.489 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_misc 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.490 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.491 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.492 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.493 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.493 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.493 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.493 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.493 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_misc 2024-01-21T04:34:43.493 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39902 2024-01-21T04:34:43.493 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_misc on pid 39902' 2024-01-21T04:34:43.493 INFO:tasks.workunit.client.0.smithi069.stdout:test api_misc on pid 39902 2024-01-21T04:34:43.513 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39902 2024-01-21T04:34:43.513 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.513 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.513 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.514 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.515 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.516 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39810/exe 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.517 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_misc_pp 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_misc_pp' 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.518 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.519 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.520 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.521 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.522 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.523 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39842/exe 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.524 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.525 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.526 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_list.log 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_list: /' 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39862/exe 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.527 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_lock.log 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_lock: /' 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_io_pp: /' 2024-01-21T04:34:43.528 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39826/exe 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.529 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_misc_pp 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.530 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.531 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.532 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.533 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.533 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_asio.log 2024-01-21T04:34:43.533 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_misc_pp 2024-01-21T04:34:43.533 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39946 2024-01-21T04:34:43.533 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_misc_pp on pid 39946' 2024-01-21T04:34:43.533 INFO:tasks.workunit.client.0.smithi069.stdout:test api_misc_pp on pid 39946 2024-01-21T04:34:43.539 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39946 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_tier_pp 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_tier_pp' 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_tier_pp 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.540 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_asio: /' 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.541 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39886/exe 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.542 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.543 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_tier_pp 2024-01-21T04:34:43.544 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39968 2024-01-21T04:34:43.545 INFO:tasks.workunit.client.0.smithi069.stdout:test api_tier_pp on pid 39968 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_tier_pp on pid 39968' 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39968 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.551 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.552 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_pool 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_pool' 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.553 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_pool 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_lock_pp: /' 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_pool 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39995 2024-01-21T04:34:43.554 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_pool on pid 39995' 2024-01-21T04:34:43.555 INFO:tasks.workunit.client.0.smithi069.stdout:test api_pool on pid 39995 2024-01-21T04:34:43.562 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=39995 2024-01-21T04:34:43.562 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.562 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.562 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_snapshots 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_snapshots' 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_snapshots 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.563 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.564 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.565 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.566 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.567 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_snapshots 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40020 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.568 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.569 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.569 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_snapshots on pid 40020' 2024-01-21T04:34:43.569 INFO:tasks.workunit.client.0.smithi069.stdout:test api_snapshots on pid 40020 2024-01-21T04:34:43.577 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40020 2024-01-21T04:34:43.577 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.577 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.577 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39902/exe 2024-01-21T04:34:43.577 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-01-21T04:34:43.577 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_snapshots_pp 2024-01-21T04:34:43.577 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.577 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_snapshots_pp' 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.578 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_snapshots_pp 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.579 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_misc: /' 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.580 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.581 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.582 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_misc.log 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.583 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_snapshots_pp 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.584 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.585 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40073 2024-01-21T04:34:43.585 INFO:tasks.workunit.client.0.smithi069.stdout:test api_snapshots_pp on pid 40073 2024-01-21T04:34:43.591 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_snapshots_pp on pid 40073' 2024-01-21T04:34:43.591 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40073 2024-01-21T04:34:43.591 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.592 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_stat 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_stat' 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_stat 2024-01-21T04:34:43.593 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.594 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.595 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.596 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.597 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_stat 2024-01-21T04:34:43.597 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.597 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.597 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40101 2024-01-21T04:34:43.597 INFO:tasks.workunit.client.0.smithi069.stdout:test api_stat on pid 40101 2024-01-21T04:34:43.605 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_stat on pid 40101' 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40101 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_stat_pp 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_stat_pp' 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.606 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_stat_pp 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.607 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.608 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.609 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39946/exe 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.610 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.611 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.612 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_stat_pp 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40114 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.613 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.614 INFO:tasks.workunit.client.0.smithi069.stdout:test api_stat_pp on pid 40114 2024-01-21T04:34:43.631 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_stat_pp on pid 40114' 2024-01-21T04:34:43.631 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40114 2024-01-21T04:34:43.631 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.631 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.631 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_watch_notify 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2024-01-21T04:34:43.632 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_watch_notify' 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39968/exe 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_misc_pp: /' 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.633 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.634 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.635 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.636 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.637 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_tier_pp: /' 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.638 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.639 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.640 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.641 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.642 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.643 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/39995/exe 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40020/exe 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.644 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_watch_notify 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_pool: /' 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_pool.log 2024-01-21T04:34:43.645 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_watch_notify 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40162 2024-01-21T04:34:43.646 INFO:tasks.workunit.client.0.smithi069.stdout:test api_watch_notify on pid 40162 2024-01-21T04:34:43.660 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_watch_notify on pid 40162' 2024-01-21T04:34:43.660 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40162 2024-01-21T04:34:43.660 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.660 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.660 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-01-21T04:34:43.660 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.660 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.660 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.661 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.662 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.663 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.664 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_watch_notify_pp 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_watch_notify_pp' 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.665 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.666 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.667 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.668 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40073/exe 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.669 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.670 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_watch_notify_pp 2024-01-21T04:34:43.671 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.672 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2024-01-21T04:34:43.672 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-01-21T04:34:43.672 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40101/exe 2024-01-21T04:34:43.672 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_watch_notify_pp 2024-01-21T04:34:43.672 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40207 2024-01-21T04:34:43.672 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_watch_notify_pp on pid 40207' 2024-01-21T04:34:43.672 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2024-01-21T04:34:43.672 INFO:tasks.workunit.client.0.smithi069.stdout:test api_watch_notify_pp on pid 40207 2024-01-21T04:34:43.673 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40207 2024-01-21T04:34:43.673 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.673 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.673 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cmd 2024-01-21T04:34:43.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-01-21T04:34:43.673 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] Global test environment set-up. 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] 12 tests from AsioRados 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.539+0000 7f9fc7239780 10 monclient: get_monmap_and_config 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.539+0000 7f9fc7239780 10 monclient: build_initial_monmap 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.539+0000 7f9fc7239780 10 monclient: monmap: 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 0 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 0.000000 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 0.000000 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 0 (unknown) 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:34:43.674 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.540+0000 7f9fc7239780 1 Processor -- start 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.540+0000 7f9fc7239780 1 -- start start 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.540+0000 7f9fc7239780 10 monclient: init 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 10 monclient: _reopen_session rank -1 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56259afa9a10 0x56259afed4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 10 monclient(hunting): picked mon.noname-a con 0x56259afa9a10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56259b07cae0 0x56259b07c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 10 monclient(hunting): picked mon.noname-c con 0x56259b07cae0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56259b07fbc0 0x56259b06b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 10 monclient(hunting): picked mon.noname-b con 0x56259b07fbc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 10 monclient(hunting): start opening mon connection 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259af6c970 con 0x56259afa9a10 2024-01-21T04:34:43.675 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 10 monclient(hunting): start opening mon connection 2024-01-21T04:34:43.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259af05da0 con 0x56259b07fbc0 2024-01-21T04:34:43.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 10 monclient(hunting): start opening mon connection 2024-01-21T04:34:43.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x56259af05a40 con 0x56259b07cae0 2024-01-21T04:34:43.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 10 monclient(hunting): _renew_subs 2024-01-21T04:34:43.676 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.541+0000 7f9fc7239780 10 monclient(hunting): authenticate will time out at 1441.882595s 2024-01-21T04:34:43.679 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cmd' 2024-01-21T04:34:43.679 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.679 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.679 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-01-21T04:34:43.679 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.680 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.681 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.682 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_snapshots: /' 2024-01-21T04:34:43.682 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cmd 2024-01-21T04:34:43.682 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cmd 2024-01-21T04:34:43.682 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40229 2024-01-21T04:34:43.682 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cmd on pid 40229' 2024-01-21T04:34:43.682 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: test api_cmd on pid 40229 2024-01-21T04:34:43.684 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40229 2024-01-21T04:34:43.684 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.684 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.684 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.684 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.684 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_stat: /' 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.685 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.686 INFO:tasks.workunit.client.0.smithi069.stdout:2024-01-21T04:34:43.544+0000 7f9fbf7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56259b07cae0 0x56259b07c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:34:43.686 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.545+0000 7f9fbf7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56259b07cae0 0x56259b07c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:54552/0 (socket says 172.21.15.69:54552) 2024-01-21T04:34:43.686 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.545+0000 7f9fbf7fe640 1 -- 172.21.15.69:0/3776066343 learned_addr learned my addr 172.21.15.69:0/3776066343 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:34:43.686 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.545+0000 7f9fbf7fe640 10 monclient(hunting): get_auth_request con 0x56259b07cae0 auth_method 0 2024-01-21T04:34:43.686 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.545+0000 7f9fbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:34:43.686 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.545+0000 7f9fbf7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:34:43.686 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.545+0000 7f9fbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:34:43.686 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.550+0000 7f9fc4e26640 1 --2- 172.21.15.69:0/3776066343 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56259b07fbc0 0x56259b06b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:34:43.686 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.555+0000 7f9fc4e26640 10 monclient(hunting): get_auth_request con 0x56259b07fbc0 auth_method 0 2024-01-21T04:34:43.686 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.555+0000 7f9fc4e26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.555+0000 7f9fc4e26640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.555+0000 7f9fc4e26640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.556+0000 7f9fc4e26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.556+0000 7f9fc4e26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.556+0000 7f9fc4e26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.556+0000 7f9fbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.556+0000 7f9fbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.556+0000 7f9fbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.558+0000 7f9fc4e26640 10 monclient(hunting): handle_auth_done global_id 4438 payload 274 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.558+0000 7f9fc4e26640 1 -- 172.21.15.69:0/3776066343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56259b07cae0 msgr2=0x56259b07c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.558+0000 7f9fc4e26640 1 --2- 172.21.15.69:0/3776066343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56259b07cae0 0x56259b07c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:34:43.687 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.558+0000 7f9fc4e26640 1 -- 172.21.15.69:0/3776066343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56259afa9a10 msgr2=0x56259afed4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T04:34:43.690 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_stat.log 2024-01-21T04:34:43.690 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cmd_pp 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cmd_pp' 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.691 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cmd_pp 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.692 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cmd_pp 2024-01-21T04:34:43.693 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40251 2024-01-21T04:34:43.693 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cmd_pp on pid 40251' 2024-01-21T04:34:43.693 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.693 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.558+0000 7f9fc4e26640 1 --2- 172.21.15.69:0/3776066343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56259afa9a10 0x56259afed4a0 unknown :-1 s=Btest api_cmd_pp on pid 40251 2024-01-21T04:34:43.698 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40251 2024-01-21T04:34:43.698 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.698 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.698 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.698 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.698 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.698 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.698 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.698 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.698 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.698 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.699 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.700 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.701 INFO:tasks.workunit.client.0.smithi069.stdout:ANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.558+0000 7f9fc4e26640 10 monclient: _finish_hunting 0 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.558+0000 7f9fc4e26640 1 monclient: found mon.noname-b 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.558+0000 7f9fc4e26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.558+0000 7f9fc4e26640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.558+0000 7f9fc4e26640 1 -- 172.21.15.69:0/3776066343 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb4005de0 con 0x56259b07fbc0 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.560+0000 7f9fc4e26640 1 --2- 172.21.15.69:0/3776066343 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56259b07fbc0 0x56259b06b2a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4051000 tx=0x7f9fb404c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29551a61ff1a7406 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 1 -- 172.21.15.69:0/3776066343 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb405c470 con 0x56259b07fbc0 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 10 monclient: dump: 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:34:43.702 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 19 (squid) 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 10 monclient: _finish_auth 0 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 10 monclient: _check_auth_tickets 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 1 -- 172.21.15.69:0/3776066343 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb405c5d0 con 0x56259b07fbc0 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 1 -- 172.21.15.69:0/3776066343 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb405c980 con 0x56259b07fbc0 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fbe7fc640 10 monclient: dump: 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:34:43.703 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:34:43.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:34:43.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 19 (squid) 2024-01-21T04:34:43.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2024-01-21T04:34:43.704 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:34:43.706 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.706 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.706 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.706 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.706 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.706 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.706 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.706 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.706 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.706 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:34:43.707 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fc7239780 5 monclient: authenticate success, global_id 4438 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fc7239780 10 monclient: get_monmap_and_config success 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.577+0000 7f9fc7239780 4 set_mon_vals no callback set 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 10 monclient: shutdown 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 1 -- 172.21.15.69:0/3776066343 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56259b07fbc0 msgr2=0x56259b06b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 1 --2- 172.21.15.69:0/3776066343 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56259b07fbc0 0x56259b06b2a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4051000 tx=0x7f9fb404c000 comp rx=0 tx=0).stop 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 1 -- 172.21.15.69:0/3776066343 shutdown_connections 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 1 --2- 172.21.15.69:0/3776066343 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56259b07fbc0 0x56259b06b2a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 1 --2- 172.21.15.69:0/3776066343 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56259b07cae0 0x56259b07c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 1 --2- 172.21.15.69:0/3776066343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56259afa9a10 0x56259afed4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 1 -- 172.21.15.69:0/3776066343 >> 172.21.15.69:0/3776066343 conn(0x56259afedc70 msgr2=0x56259afec940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 1 -- 172.21.15.69:0/3776066343 shutdown_connections 2024-01-21T04:34:43.708 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 1 -- 172.21.15.69:0/3776066343 wait complete. 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 10 monclient: build_initial_monmap 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.578+0000 7f9fc7239780 10 monclient: monmap: 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 0 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 0.000000 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 0.000000 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 0 (unknown) 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.579+0000 7f9fc7239780 1 librados: starting msgr at 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.579+0000 7f9fc7239780 1 librados: starting objecter 2024-01-21T04:34:43.709 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.579+0000 7f9fc7239780 1 Processor -- start 2024-01-21T04:34:43.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.579+0000 7f9fc7239780 1 -- start start 2024-01-21T04:34:43.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.579+0000 7f9fc7239780 1 librados: setting wanted keys 2024-01-21T04:34:43.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.579+0000 7f9fc7239780 1 librados: calling monclient init 2024-01-21T04:34:43.710 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.579+0000 7f9fc7239780 10 monclient: init 2024-01-21T04:34:43.716 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.716 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.716 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40114/exe 2024-01-21T04:34:43.716 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.716 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.716 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.717 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.718 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.719 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_stat_pp: /' 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.720 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.721 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.722 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.722 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.722 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.722 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.722 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 10 monclient: _reopen_session rank -1 2024-01-21T04:34:43.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T04:34:43.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56259afa9a10 0x56259b06e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:34:43.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 10 monclient(hunting): picked mon.noname-c con 0x56259afa9a10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T04:34:43.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56259b07cae0 0x56259b069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 10 monclient(hunting): picked mon.noname-a con 0x56259b07cae0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56259b07fbc0 0x56259b069b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 10 monclient(hunting): picked mon.noname-b con 0x56259b07fbc0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 10 monclient(hunting): start opening mon connection 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259aee5550 con 0x56259b07cae0 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 10 monclient(hunting): start opening mon connection 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b067fc0 con 0x56259b07fbc0 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 10 monclient(hunting): start opening mon connection 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x56259b0681c0 con 0x56259afa9a10 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 10 monclient(hunting): _renew_subs 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.580+0000 7f9fc7239780 10 monclient(hunting): authenticate will time out at 1441.921615s 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.584+0000 7f9fbf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56259b07cae0 0x56259b069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.585+0000 7f9fbf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56259b07cae0 0x56259b069650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48088/0 (socket says 172.21.15.69:48088) 2024-01-21T04:34:43.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.585+0000 7f9fbf7fe640 1 -- 172.21.15.69:0/1833251071 learned_addr learned my addr 172.21.15.69:0/1833251071 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T04:34:43.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.585+0000 7f9fbf7fe640 10 monclient(hunting): get_auth_request con 0x56259b07cae0 auth_method 0 2024-01-21T04:34:43.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.585+0000 7f9fbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T04:34:43.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.585+0000 7f9fbf7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T04:34:43.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.585+0000 7f9fbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T04:34:43.728 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.728 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.728 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.728 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.728 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.728 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.728 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.729 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_service 2024-01-21T04:34:43.730 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.731 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_service' 2024-01-21T04:34:43.731 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_service 2024-01-21T04:34:43.731 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.731 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40162/exe 2024-01-21T04:34:43.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_service 2024-01-21T04:34:43.731 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40318 2024-01-21T04:34:43.731 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_service on pid 40318' 2024-01-21T04:34:43.731 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.590+0000 7f9ftest api_service on pid 40318 2024-01-21T04:34:43.742 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40318 2024-01-21T04:34:43.742 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.742 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.742 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.742 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.742 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.742 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.742 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.742 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.742 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.742 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.743 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.744 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.745 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_service_pp 2024-01-21T04:34:43.746 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_service_pp' 2024-01-21T04:34:43.747 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.748 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_service_pp 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.749 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.750 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.751 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.751 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.751 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.751 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_service_pp 2024-01-21T04:34:43.751 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40349 2024-01-21T04:34:43.751 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_service_pp on pid 40349' 2024-01-21T04:34:43.751 INFO:tasks.workunit.client.0.smithi069.stdout:test api_service_pp on pid 40349 2024-01-21T04:34:43.769 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40349 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.770 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.771 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.772 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.773 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_watch_notify: /' 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.774 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40251/exe 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.775 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.776 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.777 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.778 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.779 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40229/exe 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-01-21T04:34:43.780 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.781 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.782 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.783 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_c_write_operations 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cmd.log 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cmd: /' 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_c_write_operations' 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.784 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_c_write_operations 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.785 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40318/exe 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.786 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.787 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.787 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_c_write_operations 2024-01-21T04:34:43.787 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40399 2024-01-21T04:34:43.787 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_c_write_operations on pid 40399' 2024-01-21T04:34:43.787 INFO:tasks.workunit.client.0.smithi069.stdout:test api_c_write_operations on pid 40399 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40399 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40207/exe 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.790 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_service.log 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_c_read_operations 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_c_read_operations' 2024-01-21T04:34:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.792 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.793 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.793 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.793 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.793 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_c_read_operations 2024-01-21T04:34:43.793 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_c_read_operations 2024-01-21T04:34:43.793 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40411 2024-01-21T04:34:43.793 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_c_read_operations on pid 40411' 2024-01-21T04:34:43.793 INFO:tasks.workunit.client.0.smithi069.stdout:test api_c_read_operations on pid 40411 2024-01-21T04:34:43.806 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40411 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_service: /' 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.807 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.808 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s api_cls_remote_reads 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' api_cls_remote_reads' 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.809 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo api_cls_remote_reads 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.810 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.811 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.812 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.813 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40349/exe 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.814 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.815 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.816 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.817 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.817 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_service_pp: /' 2024-01-21T04:34:43.817 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.817 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=api_cls_remote_reads 2024-01-21T04:34:43.817 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40445 2024-01-21T04:34:43.817 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test api_cls_remote_reads on pid 40445' 2024-01-21T04:34:43.817 INFO:tasks.workunit.client.0.smithi069.stdout:test api_cls_remote_reads on pid 40445 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40445 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s list_parallel 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' list_parallel' 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.826 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.827 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.828 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.829 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.830 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.831 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo list_parallel 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=list_parallel 2024-01-21T04:34:43.832 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40472 2024-01-21T04:34:43.833 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test list_parallel on pid 40472' 2024-01-21T04:34:43.835 INFO:tasks.workunit.client.0.smithi069.stdout:test list_parallel on pid 40472 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40472 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40399/exe 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.853 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.854 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.855 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.856 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.857 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.858 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.859 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40445/exe 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.860 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s open_pools_parallel 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' open_pools_parallel' 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.861 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo open_pools_parallel 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.862 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.863 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.864 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.865 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40472/exe 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.866 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.867 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=open_pools_parallel 2024-01-21T04:34:43.867 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.867 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.867 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40547 2024-01-21T04:34:43.867 INFO:tasks.workunit.client.0.smithi069.stdout:test open_pools_parallel on pid 40547 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test open_pools_parallel on pid 40547' 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40547 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s delete_pools_parallel 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' delete_pools_parallel' 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_list_parallel.log 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ list_parallel: /' 2024-01-21T04:34:43.875 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.876 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo delete_pools_parallel 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.877 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.878 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40411/exe 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.879 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.880 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=delete_pools_parallel 2024-01-21T04:34:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.882 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40571 2024-01-21T04:34:43.882 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test delete_pools_parallel on pid 40571' 2024-01-21T04:34:43.882 INFO:tasks.workunit.client.0.smithi069.stdout:test delete_pools_parallel on pid 40571 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40571 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s cls 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' cls' 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo cls 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.885 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.886 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.887 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.887 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.887 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.887 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.887 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.887 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=cls 2024-01-21T04:34:43.887 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40608 2024-01-21T04:34:43.887 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test cls on pid 40608' 2024-01-21T04:34:43.887 INFO:tasks.workunit.client.0.smithi069.stdout:test cls on pid 40608 2024-01-21T04:34:43.893 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40608 2024-01-21T04:34:43.893 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.894 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.895 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s cmd 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' cmd' 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.896 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.897 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo cmd 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.898 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=cmd 2024-01-21T04:34:43.899 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.899 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40630 2024-01-21T04:34:43.899 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test cmd on pid 40630' 2024-01-21T04:34:43.899 INFO:tasks.workunit.client.0.smithi069.stdout:test cmd on pid 40630 2024-01-21T04:34:43.909 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40630 2024-01-21T04:34:43.909 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:43.909 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.909 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-01-21T04:34:43.909 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.909 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s handler_error 2024-01-21T04:34:43.909 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.909 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' handler_error' 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.910 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.911 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo handler_error 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40547/exe 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.912 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.913 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.914 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.915 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.916 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=handler_error 2024-01-21T04:34:43.917 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40668 2024-01-21T04:34:43.918 INFO:tasks.workunit.client.0.smithi069.stdout:test handler_error on pid 40668 2024-01-21T04:34:43.930 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test handler_error on pid 40668' 2024-01-21T04:34:43.930 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40668 2024-01-21T04:34:43.930 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40571/exe 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.931 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.932 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.933 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.934 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.935 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s io 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' io' 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo io 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.936 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.937 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.938 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.939 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.940 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=io 2024-01-21T04:34:43.941 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40711 2024-01-21T04:34:43.942 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test io on pid 40711' 2024-01-21T04:34:43.942 INFO:tasks.workunit.client.0.smithi069.stdout:test io on pid 40711 2024-01-21T04:34:43.956 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40711 2024-01-21T04:34:43.956 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:43.956 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.956 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s ec_io 2024-01-21T04:34:43.956 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' ec_io' 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.957 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo ec_io 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.958 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.959 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.960 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.961 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40630/exe 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.962 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.963 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.964 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.965 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_cmd 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_cmd.log 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ cmd: /' 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40668/exe 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.966 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:43.967 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40608/exe 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_handler_error 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_handler_error.log 2024-01-21T04:34:43.968 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ handler_error: /' 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_cls.log 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=ec_io 2024-01-21T04:34:43.969 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40754 2024-01-21T04:34:43.970 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test ec_io on pid 40754' 2024-01-21T04:34:43.970 INFO:tasks.workunit.client.0.smithi069.stdout:test ec_io on pid 40754 2024-01-21T04:34:43.975 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40754 2024-01-21T04:34:43.975 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:43.975 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ cls: /' 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.976 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_cls 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s list 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' list' 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:43.977 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo list 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.978 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.979 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.980 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.980 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=list 2024-01-21T04:34:43.980 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40770 2024-01-21T04:34:43.980 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test list on pid 40770' 2024-01-21T04:34:43.980 INFO:tasks.workunit.client.0.smithi069.stdout:test list on pid 40770 2024-01-21T04:34:43.981 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40770 2024-01-21T04:34:43.981 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:43.981 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:43.981 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:43.981 INFO:tasks.workunit.client.0.smithi069.stdout:bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T04:34:43.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.590+0000 7f9fbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T04:34:43.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.590+0000 7f9fbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T04:34:43.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.596+0000 7f9fbf7fe640 10 monclient(hunting): handle_auth_done global_id 4407 payload 824 2024-01-21T04:34:43.981 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.596+0000 7f9fbf7fe640 1 -- 172.21.15.69:0/1833251071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56259afa9a10 msgr2=0x56259b06e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.596+0000 7f9fbf7fe640 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x56259afa9a10 0x56259b06e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.596+0000 7f9fbf7fe640 1 -- 172.21.15.69:0/1833251071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56259b07fbc0 msgr2=0x56259b069b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.596+0000 7f9fbf7fe640 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56259b07fbc0 0x56259b069b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.596+0000 7f9fbf7fe640 10 monclient: _finish_hunting 0 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.596+0000 7f9fbf7fe640 1 monclient: found mon.noname-a 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.596+0000 7f9fbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.596+0000 7f9fbf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.596+0000 7f9fbf7fe640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56259af2bb90 con 0x56259b07cae0 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.702+0000 7f9fbf7fe640 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56259b07cae0 0x56259b069650 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9fb800d4d0 tx=0x7f9fb800d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.702+0000 7f9fbd7fa640 10 client.?.objecter ms_handle_connect 0x56259b07cae0 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.702+0000 7f9fbd7fa640 10 client.?.objecter resend_mon_ops 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.826+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8011600 con 0x56259b07cae0 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.826+0000 7f9fbd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.826+0000 7f9fbd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T04:34:43.982 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.826+0000 7f9fbd7fa640 10 monclient: dump: 2024-01-21T04:34:43.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2024-01-21T04:34:43.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:34:43.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:34:43.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:34:43.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 19 (squid) 2024-01-21T04:34:43.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2024-01-21T04:34:43.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:34:43.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:34:43.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:34:43.983 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.990 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.991 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.992 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s ec_list 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' ec_list' 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.993 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.994 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo ec_list 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:43.995 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:43.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=ec_list 2024-01-21T04:34:43.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40811 2024-01-21T04:34:43.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test ec_list on pid 40811' 2024-01-21T04:34:43.996 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fbd7fa640 10test ec_list on pid 40811 2024-01-21T04:34:44.007 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40811 2024-01-21T04:34:44.007 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:44.007 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:44.007 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:44.007 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:44.007 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:44.007 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.007 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:44.007 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s misc 2024-01-21T04:34:44.008 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' misc' 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:44.009 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.010 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40711/exe 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.011 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.012 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_io 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_io.log 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:44.013 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ io: /' 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:44.014 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo misc 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:44.015 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=misc 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40882 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test misc on pid 40882' 2024-01-21T04:34:44.016 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.017 INFO:tasks.workunit.client.0.smithi069.stdout:test misc on pid 40882 2024-01-21T04:34:44.024 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:44.024 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40882 2024-01-21T04:34:44.024 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.024 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:44.024 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.025 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s pool 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' pool' 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.026 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40770/exe 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:44.027 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo pool 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40754/exe 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:44.028 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_list 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ list: /' 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_list.log 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.029 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=pool 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40907 2024-01-21T04:34:44.030 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test pool on pid 40907' 2024-01-21T04:34:44.031 INFO:tasks.workunit.client.0.smithi069.stdout:test pool on pid 40907 2024-01-21T04:34:44.036 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40907 2024-01-21T04:34:44.036 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:44.036 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:44.036 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:44.036 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:44.036 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:44.036 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s read_operations 2024-01-21T04:34:44.036 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' read_operations' 2024-01-21T04:34:44.036 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_ec_io.log 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_ec_io 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:44.037 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:44.038 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.039 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ ec_io: /' 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo read_operations 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=read_operations 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40923 2024-01-21T04:34:44.040 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test read_operations on pid 40923' 2024-01-21T04:34:44.041 INFO:tasks.workunit.client.0.smithi069.stdout:test read_operations on pid 40923 2024-01-21T04:34:44.046 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40923 2024-01-21T04:34:44.046 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:44.046 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:44.046 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s snapshots 2024-01-21T04:34:44.046 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' snapshots' 2024-01-21T04:34:44.046 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:44.046 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.047 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo snapshots 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.048 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:44.049 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.050 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:44.051 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.052 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stdout: monclient: _finish_auth 0 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fbd7fa640 10 monclient: _check_auth_tickets 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb8011760 con 0x56259b07cae0 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fbd7fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8011ad0 con 0x56259b07cae0 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fbd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fbd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fbd7fa640 10 monclient: dump: 2024-01-21T04:34:44.053 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: epoch 1 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: created 2024-01-21T04:31:59.217447+0000 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: min_mon_release 19 (squid) 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: election_strategy: 1 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fc7239780 5 monclient: authenticate success, global_id 4407 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fc7239780 10 monclient: _renew_subs 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fc7239780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56259af11ba0 con 0x56259b07cae0 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fc7239780 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:34:44.054 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fc7239780 10 monclient: _renew_subs 2024-01-21T04:34:44.055 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fc7239780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:34:44.055 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56259af04d50 con 0x56259b07cae0 2024-01-21T04:34:44.055 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fc7239780 1 librados: init done 2024-01-21T04:34:44.055 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fc7239780 10 librados: wait_for_osdmap waiting 2024-01-21T04:34:44.055 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.827+0000 7f9fbdffb640 4 set_mon_vals no callback set 2024-01-21T04:34:44.055 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.879+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f9fb8011c30 con 0x56259b07cae0 2024-01-21T04:34:44.055 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.879+0000 7f9fbd7fa640 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f9fac06aad0 0x7f9fac06cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:34:44.057 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:44.057 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:44.057 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:44.057 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.058 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:44.059 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=snapshots 2024-01-21T04:34:44.059 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:44.059 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40939 2024-01-21T04:34:44.059 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test snapshots on pid 40939' 2024-01-21T04:34:44.059 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.880+0000 7f9fbffff640 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f9fac06aad0 0x7f9fac06cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supptest snapshots on pid 40939 2024-01-21T04:34:44.063 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40939 2024-01-21T04:34:44.063 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:44.063 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:44.063 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40811/exe 2024-01-21T04:34:44.063 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s watch_notify 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' watch_notify' 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo watch_notify 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:44.064 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:44.065 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=watch_notify 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40952 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test watch_notify on pid 40952' 2024-01-21T04:34:44.066 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-01-21T04:34:44.067 INFO:tasks.workunit.client.0.smithi069.stdout:test watch_notify on pid 40952 2024-01-21T04:34:44.072 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40952 2024-01-21T04:34:44.072 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.072 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:44.073 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:++ printf %25s write_operations 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:44.074 INFO:tasks.workunit.client.0.smithi069.stderr:+ r=' write_operations' 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_ec_list 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_ec_list.log 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:++ awk '{print $1}' 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.075 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ echo write_operations 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.076 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.077 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:44.077 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.077 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:44.077 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.077 INFO:tasks.workunit.client.0.smithi069.stderr:+ ff=write_operations 2024-01-21T04:34:44.077 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40967 2024-01-21T04:34:44.077 INFO:tasks.workunit.client.0.smithi069.stderr:+ echo 'test write_operations on pid 40967' 2024-01-21T04:34:44.077 INFO:tasks.workunit.client.0.smithi069.stdout:test write_operations on pid 40967 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:+ pids[$f]=40967 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:+ ret=0 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' 1 -eq 1 ']' 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39968 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39968 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ ec_list: /' 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.113 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -f /etc/bashrc ']' 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:+ . /etc/bashrc 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -z '' ']' 2024-01-21T04:34:44.114 INFO:tasks.workunit.client.0.smithi069.stderr:++ BASHRCSOURCED=Y 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:++ shopt -q login_shell 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:+++ umask 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' 0022 -eq 0 ']' 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:++ SHELL=/bin/bash 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/bash_completion.sh 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-01-21T04:34:44.115 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/cobbler.sh 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorgrep.sh 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.116 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'grep=grep --color=auto' 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'egrep=egrep --color=auto' 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-01-21T04:34:44.117 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorls.sh 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' '!' -t 0 ']' 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:+++ return 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export S_COLORS=auto 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:+++ S_COLORS=auto 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:44.118 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:44.119 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:44.120 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.121 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:44.122 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ /usr/libexec/grepconf.sh -c 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:44.123 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:44.124 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.125 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40923/exe 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:44.126 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:44.127 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/debuginfod.sh 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-01-21T04:34:44.128 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/gawk.sh 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/lang.sh 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.129 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/locale 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:44.130 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:44.131 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.132 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG_backup=en_US.UTF-8 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /etc/locale.conf ']' 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -x /usr/bin/sed ']' 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:44.133 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-01-21T04:34:44.134 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_read_operations.log 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ read_operations: /' 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:44.135 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:44.136 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40907/exe 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:++++ export LANG=en_US.UTF-8 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:++++ LANG=en_US.UTF-8 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:44.137 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_read_operations 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40882/exe 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ LANG=en_US.UTF-8 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ unset LANG_backup config 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n '' ']' 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' dumb = linux ']' 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/less.sh 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-01-21T04:34:44.138 INFO:tasks.workunit.client.0.smithi069.stderr:++ for i in /etc/profile.d/*.sh 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ '[' '' ']' 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ . /etc/profile.d/which2.sh 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40967/exe 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:44.139 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_misc 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_misc.log 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:44.140 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_write_operations 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ write_operations: /' 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40952/exe 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ misc: /' 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:44.141 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_write_operations.log 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_pool 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_pool.log 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ pool: /' 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+++++ readlink /proc/40939/exe 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:++++ basename /usr/bin/bash 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ watch_notify: /' 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_watch_notify 2024-01-21T04:34:44.142 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_declare='declare -f' 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:+++ which_opt=-f 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export which_declare 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:+++ export -f which 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset i 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:++ unset -f pathmunge 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:+ export PATH 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:+ unset rc 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:+ sed 's/^/ snapshots: /' 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:+ tee ceph_test_neorados_snapshots.log 2024-01-21T04:34:44.143 INFO:tasks.workunit.client.0.smithi069.stderr:+ ceph_test_neorados_snapshots 2024-01-21T04:34:46.291 INFO:tasks.workunit.client.0.smithi069.stdout:orted=3 required=0 2024-01-21T04:34:46.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.880+0000 7f9fbffff640 10 monclient: get_auth_request con 0x7f9fac06aad0 auth_method 0 2024-01-21T04:34:46.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.880+0000 7f9fbffff640 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f9fac06aad0 0x7f9fac06cf90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0005f80 tx=0x7f9fb0005530 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:34:46.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.880+0000 7f9fbd7fa640 10 client.4407.objecter ms_handle_connect 0x7f9fac06aad0 2024-01-21T04:34:46.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.893+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9fb8018310 con 0x56259b07cae0 2024-01-21T04:34:46.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.893+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(26..26 src has 1..26) v4 2024-01-21T04:34:46.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.893+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [26,26] > 0 2024-01-21T04:34:46.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.893+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding full epoch 26 2024-01-21T04:34:46.291 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.893+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.893+0000 7f9fc7239780 10 librados: wait_for_osdmap done waiting 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.893+0000 7f9fc7239780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:43.893+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x56259af65800 con 0x56259b07cae0 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:44.122+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fb8053650 con 0x56259b07cae0 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:44.122+0000 7f9fbd7fa640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:44.122+0000 7f9fbe7fc640 10 client.4407.objecter _get_latest_version latest 26, have it 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:44.122+0000 7f9fc7239780 10 client.4407.objecter create_pool name=ceph_test_rados_api_asio 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:44.122+0000 7f9fc7239780 10 client.4407.objecter pool_op_submit 1 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:44.122+0000 7f9fc7239780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:44.122+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x56259af76a00 con 0x56259b07cae0 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.290+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v28) v1 ==== 43+0+0 (secure 0 0 0) 0x7f9fb8015030 con 0x56259b07cae0 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.290+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 pool_op_reply(tid 1 (0) Success v28) v1 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.290+0000 7f9fbd7fa640 10 client.4407.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v28) v1 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.290+0000 7f9fbd7fa640 10 client.4407.objecter have request 1 at 0x56259b069410 Op: create 2024-01-21T04:34:46.292 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.290+0000 7f9fbd7fa640 20 client.4407.objecter waiting for client to reach epoch 28 before calling back 2024-01-21T04:34:46.293 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.290+0000 7f9fbd7fa640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:34:46.432 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.290+0000 7f9fbd7fa640 10 monclient: _renew_subs 2024-01-21T04:34:46.432 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.290+0000 7f9fbd7fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:34:46.432 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.290+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=27}) v3 -- 0x7f9fac070810 con 0x56259b07cae0 2024-01-21T04:34:46.432 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.290+0000 7f9fbd7fa640 10 client.4407.objecter done 2024-01-21T04:34:46.432 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 8 ==== osd_map(27..28 src has 1..28) v4 ==== 15340+0+0 (secure 0 0 0) 0x7f9fb8057a10 con 0x56259b07cae0 2024-01-21T04:34:46.432 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(27..28 src has 1..28) v4 2024-01-21T04:34:46.432 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [27,28] > 26 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 27 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 28 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 10 librados: set snap read head -> 1 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 10 librados: writefull oid=exist nspace= 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b12af10 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 28 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x56259b068c80 0x56259b12d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 2 osd.1 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:34:46.433 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 2 2024-01-21T04:34:46.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 15 client.4407.objecter _send_op 2 to 12.1 on osd.1 2024-01-21T04:34:46.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) v8 -- 0x56259b06a0d0 con 0x56259b068c80 2024-01-21T04:34:46.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:34:46.434 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.430+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:34:47.237 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:15.144 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:47.249 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:15.156 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:47.303 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:15.210 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:47.312 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:15.216 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:47.350 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:15.257 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:47.366 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:15.273 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:47.395 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:02:15.270 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:47.400 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:02:15.275 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:47.484 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:02:15.359 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:34:54.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.431+0000 7f9fc4e26640 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x56259b068c80 0x56259b12d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_paylo api_service: [==========] Running 4 tests from 1 test suite. 2024-01-21T04:34:54.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] Global test environment set-up. 2024-01-21T04:34:54.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] 4 tests from LibRadosService 2024-01-21T04:34:54.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-01-21T04:34:54.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5338 ms) 2024-01-21T04:34:54.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-01-21T04:34:54.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.RegisterLate (94 ms) 2024-01-21T04:34:54.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-01-21T04:34:54.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2024-01-21T04:34:54.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:34:54.800 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_WARN 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 32 pool(s) do not have an application enabled 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 71s), 8 in (since 2m) 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 37 pools, 1129 pgs 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 2 objects, 449 KiB 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 98.494% pgs unknown 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 0.709% pgs not active 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 1112 unknown 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 9 active+clean 2024-01-21T04:34:54.801 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 8 creating+peering 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_WARN 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 19 pool(s) do not have an application enabled 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 74s), 8 in (since 2m) 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 38 pools, 1113 pgs 2024-01-21T04:34:54.802 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 2 objects, 449 KiB 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 61.725% pgs unknown 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 37.287% pgs not active 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 687 unknown 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 397 creating+peering 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 18 creating+activating 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 11 active+clean 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: cluster: 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: id: 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: health: HEALTH_WARN 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 18 pool(s) do not have an application enabled 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: services: 2024-01-21T04:34:54.803 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-01-21T04:34:54.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-01-21T04:34:54.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: osd: 8 osds: 8 up (since 76s), 8 in (since 2m) 2024-01-21T04:34:54.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-01-21T04:34:54.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-01-21T04:34:54.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:34:54.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: data: 2024-01-21T04:34:54.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pools: 39 pools, 1145 pgs 2024-01-21T04:34:54.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: objects: 2 objects, 449 KiB 2024-01-21T04:34:54.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-01-21T04:34:54.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: pgs: 21.659% pgs unknown 2024-01-21T04:34:54.804 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 76.681% pgs not active 2024-01-21T04:34:55.582 INFO:tasks.workunit.client.0.smithi069.stdout: ad supported=3 required=0 2024-01-21T04:34:55.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.432+0000 7f9fc4e26640 10 monclient: get_auth_request con 0x56259b068c80 auth_method 0 2024-01-21T04:34:55.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.440+0000 7f9fbd7fa640 10 client.4407.objecter ms_handle_connect 0x56259b068c80 2024-01-21T04:34:55.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.440+0000 7f9fc4e26640 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x56259b068c80 0x56259b12d8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T04:34:55.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.580+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:55.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.580+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:55.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.580+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:55.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:46.580+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:47.580+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:47.580+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:47.580+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:48.580+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:48.580+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:48.580+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:48.828+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:49.580+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:49.580+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:49.580+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:50.580+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:50.580+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:50.580+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:55.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:51.580+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:51.580+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:51.580+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:52.580+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:52.580+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:52.580+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:53.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:53.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:53.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:53.829+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:54.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:54.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:54.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:55.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:55.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:55.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:55.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:58.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:55.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:58.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:56.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:56.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:56.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:56.581+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:57.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:57.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:57.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.830+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.830+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:34:58.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.830+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.830+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.830+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.830+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=29}) v3 -- 0x7f9f90002f90 con 0x56259b07cae0 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.830+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003320 con 0x56259b068c80 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 9 ==== osd_map(29..36 src has 1..36) v4 ==== 20879+0+0 (secure 0 0 0) 0x7f9fb805b460 con 0x56259b07cae0 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(29..36 src has 1..36) v4 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [29,36] > 28 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 29 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 29 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 30 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:34:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 30 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:34:58.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:34:58.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 31 2024-01-21T04:34:58.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:01.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 31 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:01.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:01.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 32 2024-01-21T04:35:01.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 32 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 33 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 33 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 34 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 34 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 35 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 35 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 36 2024-01-21T04:35:01.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:01.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 36 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:01.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:01.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:01.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:58.834+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:01.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:59.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:01.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:59.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:01.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:34:59.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:01.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:00.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:01.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:00.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:01.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:00.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:01.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:01.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:03.842 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:01.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:01.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:02.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:02.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:02.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.831+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.831+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.831+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.831+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.831+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:03.843 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.831+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=37}) v3 -- 0x7f9f900034e0 con 0x56259b07cae0 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.831+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f900036d0 con 0x56259b068c80 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.841+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 10 ==== osd_map(37..40 src has 1..40) v4 ==== 4392+0+0 (secure 0 0 0) 0x7f9fb805f610 con 0x56259b07cae0 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.841+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(37..40 src has 1..40) v4 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.841+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [37,40] > 36 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.841+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 37 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.841+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.841+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 37 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.841+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.841+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 38 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 38 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 39 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:03.844 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 39 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:03.845 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:09.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 40 2024-01-21T04:35:09.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:09.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 40 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:09.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:09.065 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:03.842+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:04.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:04.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:04.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:05.582+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:05.582+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:05.582+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:06.582+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:06.582+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:06.582+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:06.582+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:07.582+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:07.582+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:07.582+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:09.066 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:08.581+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:08.581+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:08.581+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:08.831+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:08.831+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:08.831+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:08.831+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:08.831+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:08.831+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=41}) v3 -- 0x7f9f90003860 con 0x56259b07cae0 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:08.831+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f900039f0 con 0x56259b068c80 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 11 ==== osd_map(41..44 src has 1..44) v4 ==== 2878+0+0 (secure 0 0 0) 0x7f9fb80572a0 con 0x56259b07cae0 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(41..44 src has 1..44) v4 2024-01-21T04:35:09.067 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [41,44] > 40 2024-01-21T04:35:09.370 INFO:tasks.workunit.client.0.smithi069.stdout: pool: Running main() from gmock_main.cc 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [----------] Global test environment set-up. 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolList (4393 ms) 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4246 ms) 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2861 ms) 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (5472 ms) 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (4143 ms) 2024-01-21T04:35:09.371 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-01-21T04:35:09.372 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (4311 ms) 2024-01-21T04:35:09.372 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [----------] 6 tests from NeoRadosPools (25426 ms total) 2024-01-21T04:35:09.372 INFO:tasks.workunit.client.0.smithi069.stdout: pool: 2024-01-21T04:35:09.372 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [----------] Global test environment tear-down 2024-01-21T04:35:09.372 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [==========] 6 tests from 1 test suite ran. (25426 ms total) 2024-01-21T04:35:09.372 INFO:tasks.workunit.client.0.smithi069.stdout: pool: [ PASSED ] 6 tests. 2024-01-21T04:35:12.295 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:40.201 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:12.357 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:40.263 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:12.421 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:40.328 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:12.422 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:02:40.297 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:12.432 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:02:40.307 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:12.433 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:40.340 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:12.684 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:02:40.591 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:12.702 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:02:40.609 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:12.717 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:02:40.592 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:13.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 41 2024-01-21T04:35:13.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:13.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 41 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:13.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:13.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 42 2024-01-21T04:35:13.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:13.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 42 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:13.833 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 43 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 43 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 44 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 44 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.062+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.582+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.582+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:09.582+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:10.582+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:10.582+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:10.582+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:11.582+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:11.582+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:11.582+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:12.582+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:12.582+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:12.582+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.582+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.582+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.582+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:14.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_asi api_service: 774 creating+peering 2024-01-21T04:35:14.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 248 unknown 2024-01-21T04:35:14.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 104 creating+activating 2024-01-21T04:35:14.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 19 active+clean 2024-01-21T04:35:14.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:35:14.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:35:14.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5716 ms) 2024-01-21T04:35:14.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ RUN ] LibRadosService.Status 2024-01-21T04:35:14.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ OK ] LibRadosService.Status (20065 ms) 2024-01-21T04:35:14.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] 4 tests from LibRadosService (31213 ms total) 2024-01-21T04:35:14.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: 2024-01-21T04:35:14.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [----------] Global test environment tear-down 2024-01-21T04:35:14.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31214 ms total) 2024-01-21T04:35:14.910 INFO:tasks.workunit.client.0.smithi069.stdout: api_service: [ PASSED ] 4 tests. 2024-01-21T04:35:15.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-01-21T04:35:15.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] Global test environment set-up. 2024-01-21T04:35:15.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-01-21T04:35:15.541 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5287 ms) 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (969 ms) 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20309 ms) 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: attempt 0 of 20 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5261 ms) 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (31826 ms total) 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [----------] Global test environment tear-down 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (31826 ms total) 2024-01-21T04:35:15.542 INFO:tasks.workunit.client.0.smithi069.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-01-21T04:35:15.583 INFO:tasks.workunit.client.0.smithi069.stdout:o: 2024-01-21T04:35:13.832+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:15.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.832+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:15.583 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.832+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.832+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.832+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.832+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=45}) v3 -- 0x7f9f90003b80 con 0x56259b07cae0 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.832+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f900039f0 con 0x56259b068c80 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.999+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 12 ==== osd_map(45..47 src has 1..47) v4 ==== 3048+0+0 (secure 0 0 0) 0x7f9fb8053a20 con 0x56259b07cae0 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.999+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(45..47 src has 1..47) v4 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.999+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [45,47] > 44 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:13.999+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 45 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 45 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 46 2024-01-21T04:35:15.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 46 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 47 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 47 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.000+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.582+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.582+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:14.582+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:15.582+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:15.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:15.582+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:17.338 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:35:17.330+0000 12a57640 -1 osd.6 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4439.0:2 32.1 32:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:17.353 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:17.343+0000 ea4f640 -1 osd.1 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:18.472 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:18.467+0000 ea4f640 -1 osd.1 49 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4407.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:18.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:15.582+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:18.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:16.582+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:18.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:16.582+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:18.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:16.582+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:18.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:16.583+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:35:18.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:17.583+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:18.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:17.583+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:18.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:17.583+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:18.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.583+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:18.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.583+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.583+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.833+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.833+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.833+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.833+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.833+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.833+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=48}) v3 -- 0x7f9f90003b80 con 0x56259b07cae0 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.833+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003d60 con 0x56259b068c80 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 13 ==== osd_map(48..50 src has 1..50) v4 ==== 1203+0+0 (secure 0 0 0) 0x7f9fb805b100 con 0x56259b07cae0 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(48..50 src has 1..50) v4 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [48,50] > 47 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 48 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:18.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 48 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:18.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:18.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 49 2024-01-21T04:35:18.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:18.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 49 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:18.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:18.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 50 2024-01-21T04:35:18.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:18.845 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:35:18.717+0000 12a57640 -1 osd.6 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4439.0:2 32.1 32:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:19.478 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:19.475+0000 ea4f640 -1 osd.1 49 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4407.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:19.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 20 api_misc: [==========] Running 12 tests from 4 test suites. 2024-01-21T04:35:19.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] Global test environment set-up. 2024-01-21T04:35:19.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-01-21T04:35:19.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-01-21T04:35:19.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-01-21T04:35:19.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-01-21T04:35:19.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2024-01-21T04:35:19.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-01-21T04:35:19.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-01-21T04:35:19.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: unable to get monitor info from DNS SRV with service name: 2024-01-21T04:34:43.761+0000 7f50bea0f780 -1 failed for service _ceph-mon._tcp 2024-01-21T04:35:19.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: ceph-mon 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2024-01-21T04:34:43.761+0000 7f50bea0f780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (224 ms) 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2012 ms) 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2236 ms total) 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562ceb9b20d0 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebc00990 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562ceb9b6170 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebb341a0 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebb0a1e0 2024-01-21T04:35:19.783 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebb49b30 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebb4a520 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebb4afd0 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd372c0 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebb4b350 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd38250 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd37040 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd39160 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd38040 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd3a0e0 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd38f20 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd3ad20 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd43f70 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd39e00 2024-01-21T04:35:19.784 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd44df0 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd43d70 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd3b190 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd44b90 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd3c030 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd3af90 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd3cef0 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd3bdc0 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd3dd70 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd3ccd0 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 0x562cebd3ec30 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: started 30 aios 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562ceb9b20d0 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebc00990 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562ceb9b6170 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebb341a0 2024-01-21T04:35:19.785 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebb0a1e0 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebb49b30 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebb4a520 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebb4afd0 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd372c0 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebb4b350 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd38250 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd37040 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd39160 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd38040 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd3a0e0 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd38f20 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd3ad20 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd43f70 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd39e00 2024-01-21T04:35:19.786 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd44df0 2024-01-21T04:35:19.957 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:35:19.898+0000 ea4f640 -1 osd.2 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 13.1b 13.ead06fbb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-01-21T04:35:20.093 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:35:20.047+0000 12a57640 -1 osd.6 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4439.0:2 32.1 32:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:20.731 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:20.715+0000 ea4f640 -1 osd.1 50 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:21.311 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:35:21.057+0000 ea4f640 -1 osd.2 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 13.1b 13.ead06fbb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-01-21T04:35:21.406 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T04:35:21.392+0000 ea4f640 -1 osd.5 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4489.0:2 21.2 21:416182a0:test-rados-api-smithi069-40131-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:21.799 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:35:21.794+0000 12a57640 -1 osd.6 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4439.0:2 32.1 32:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:22.101 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:21.926+0000 ea4f640 -1 osd.1 51 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:22.213 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:35:22.139+0000 ea4f640 -1 osd.2 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 13.1b 13:ddf60b57:test-rados-api-smithi069-40219-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-01-21T04:35:22.462 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T04:35:22.452+0000 ea4f640 -1 osd.5 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4489.0:2 21.2 21:416182a0:test-rados-api-smithi069-40131-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:22.923 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:22.915+0000 ea4f640 -1 osd.1 51 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:22.994 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:35:22.980+0000 12a57640 -1 osd.6 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4439.0:2 32.1 32:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:23.506 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T04:35:23.501+0000 ea4f640 -1 osd.5 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4489.0:2 21.2 21:416182a0:test-rados-api-smithi069-40131-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:23.722 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:35:23.545+0000 ea4f640 -1 osd.2 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 13.1b 13:ddf60b57:test-rados-api-smithi069-40219-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-01-21T04:35:24.007 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:23.985+0000 ea4f640 -1 osd.1 52 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:24.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x56 client.4407.objecter _calc_target epoch 50 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:24.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:24.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:24.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:18.836+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:19.583+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:19.583+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:19.583+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:20.583+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:20.583+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:20.583+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:21.583+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:21.583+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:21.583+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:22.583+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:22.583+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:22.583+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:24.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:23.583+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:23.583+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:23.583+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:23.834+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:23.834+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:23.834+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:23.834+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:23.834+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:23.834+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=51}) v3 -- 0x7f9f90003b80 con 0x56259b07cae0 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:23.834+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003d60 con 0x56259b068c80 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 14 ==== osd_map(51..53 src has 1..53) v4 ==== 1243+0+0 (secure 0 0 0) 0x7f9fb8080d90 con 0x56259b07cae0 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(51..53 src has 1..53) v4 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [51,53] > 50 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 51 2024-01-21T04:35:24.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:24.013 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 51 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:24.049 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:35:24.045+0000 12a57640 -1 osd.6 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4439.0:2 32.1 32:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:24.562 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:35:24.555+0000 ea4f640 -1 osd.2 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 13.1b 13:ddf60b57:test-rados-api-smithi069-40219-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-01-21T04:35:24.639 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T04:35:24.624+0000 ea4f640 -1 osd.5 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4489.0:2 21.2 21:416182a0:test-rados-api-smithi069-40131-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:25.075 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:25.058+0000 ea4f640 -1 osd.1 52 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:25.176 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:35:25.151+0000 12a57640 -1 osd.6 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4439.0:2 32.1 32:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:25.615 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:35:25.604+0000 ea4f640 -1 osd.2 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 13.1b 13:ddf60b57:test-rados-api-smithi069-40219-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-01-21T04:35:25.868 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T04:35:25.856+0000 ea4f640 -1 osd.5 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4489.0:2 21.2 21:416182a0:test-rados-api-smithi069-40131-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:26.195 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:35:26.179+0000 12a57640 -1 osd.6 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4439.0:2 32.1 32:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:26.203 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:26.117+0000 ea4f640 -1 osd.1 53 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:26.955 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:35:26.848+0000 ea4f640 -1 osd.2 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 13.1b 13:ddf60b57:test-rados-api-smithi069-40219-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-01-21T04:35:27.229 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T04:35:27.219+0000 ea4f640 -1 osd.5 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4489.0:2 21.2 21:416182a0:test-rados-api-smithi069-40131-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:27.434 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:27.306+0000 ea4f640 -1 osd.1 55 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:27.497 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:35:27.470+0000 12a57640 -1 osd.6 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4439.0:2 32.1 32:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:27.896 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:35:27.887+0000 ea4f640 -1 osd.2 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 13.1b 13:ddf60b57:test-rados-api-smithi069-40219-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-01-21T04:35:28.378 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:28.368+0000 ea4f640 -1 osd.1 55 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:28.487 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T04:35:28.389+0000 ea4f640 -1 osd.5 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4489.0:2 21.2 21:416182a0:test-rados-api-smithi069-40131-2::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:28.550 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:35:28.518+0000 12a57640 -1 osd.6 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4439.0:2 32.1 32:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 52 2024-01-21T04:35:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 52 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 53 2024-01-21T04:35:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 53 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:28.836 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.009+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.583+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.583+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:24.583+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:25.429+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 1 ==== osd_map(29..52 src has 1..53) v4 ==== 32816+0+0 (crc 0 0 0) 0x7f9fb406d040 con 0x56259b068c80 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:25.429+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(29..52 src has 1..53) v4 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:25.429+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map ignoring epochs [29,52] <= 53 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:25.429+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:25.429+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:25.583+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:25.583+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:25.583+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:26.583+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:28.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:26.583+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:28.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:26.583+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:28.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:26.583+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:35:28.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:27.584+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:28.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:27.584+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:28.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:27.584+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:28.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.584+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:28.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.584+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:28.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.584+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:29.167 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:35:29.113+0000 ea4f640 -1 osd.2 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 13.1b 13:ddf60b57:test-rados-api-smithi069-40219-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-01-21T04:35:29.443 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:29.429+0000 ea4f640 -1 osd.1 56 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:29.533 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:35:29.519+0000 12a57640 -1 osd.6 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4439.0:2 32.1 32:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:29.584 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.835+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.835+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.835+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.835+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.835+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.835+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=54}) v3 -- 0x7f9f90003b80 con 0x56259b07cae0 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.835+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003d60 con 0x56259b068c80 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 15 ==== osd_map(54..57 src has 1..57) v4 ==== 3269+0+0 (secure 0 0 0) 0x7f9fb8057570 con 0x56259b07cae0 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(54..57 src has 1..57) v4 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [54,57] > 53 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 54 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 54 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:29.585 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 55 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 55 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 56 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 56 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 57 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 57 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:28.838+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:29.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:29.584+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:30.152 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:35:30.141+0000 ea4f640 -1 osd.2 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4464.0:2 13.1b 13:ddf60b57:test-rados-api-smithi069-40219-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-01-21T04:35:30.688 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:30.508+0000 ea4f640 -1 osd.1 57 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:31.521 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:31.511+0000 ea4f640 -1 osd.1 57 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:32.424 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:00.331 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:32.431 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:00.338 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:32.505 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:00.412 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:32.513 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:00.420 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:32.647 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:00.554 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:32.699 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:00.604 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:32.705 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:03:00.580 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:32.711 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:03:00.586 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:32.934 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:03:00.809 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:32.949 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:32.903+0000 ea4f640 -1 osd.1 58 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:29.584+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:29.584+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:30.584+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:30.584+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:30.584+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:31.584+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:31.584+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:31.584+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:32.584+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:32.584+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:32.584+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:33.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.584+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.584+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.584+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.836+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.836+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.836+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.836+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.836+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.836+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=58}) v3 -- 0x7f9f90003b80 con 0x56259b07cae0 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.836+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003d60 con 0x56259b068c80 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 16 ==== osd_map(58..61 src has 1..61) v4 ==== 3636+0+0 (secure 0 0 0) 0x7f9fb8057570 con 0x56259b07cae0 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(58..61 src has 1..61) v4 2024-01-21T04:35:33.896 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [58,61] > 57 2024-01-21T04:35:33.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 58 2024-01-21T04:35:33.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:33.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 58 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:33.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:33.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 59 2024-01-21T04:35:33.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:33.897 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 59 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:34.014 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:33.954+0000 ea4f640 -1 osd.1 59 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:35.261 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:35.133+0000 ea4f640 -1 osd.1 60 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:36.255 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:36.247+0000 ea4f640 -1 osd.1 60 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:37.207 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:37.123+0000 12a57640 -1 osd.3 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:37.566 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:37.552+0000 ea4f640 -1 osd.1 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:38.419 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:38.332+0000 12a57640 -1 osd.3 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:38.749 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:38.718+0000 ea4f640 -1 osd.1 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:38.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:38.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 60 2024-01-21T04:35:38.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:38.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 60 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:38.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:38.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 61 2024-01-21T04:35:38.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:38.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 61 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:38.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:38.838 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:33.894+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:34.584+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:34.584+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:34.584+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:35.584+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:35.584+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:35.584+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:36.584+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:36.584+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:36.584+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:36.584+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:37.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:37.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:37.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:38.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:38.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:38.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:38.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.837+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:38.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.837+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:38.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.837+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:38.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.837+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:38.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.837+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:38.840 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.837+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=62}) v3 -- 0x7f9f90003b80 con 0x56259b07cae0 2024-01-21T04:35:39.512 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:39.473+0000 12a57640 -1 osd.3 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:40.004 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:39.988+0000 ea4f640 -1 osd.1 63 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:40.583 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:40.574+0000 12a57640 -1 osd.3 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:40.593 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:40.508+0000 ea4f640 -1 osd.0 64 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4457.0:2 27.6 27:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:41.624 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:41.535+0000 ea4f640 -1 osd.0 65 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4457.0:2 27.6 27:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:41.639 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:41.620+0000 12a57640 -1 osd.3 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:41.788 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:41.681+0000 ea4f640 -1 osd.1 65 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:42.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.837+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003ef0 con 0x56259b068c80 2024-01-21T04:35:42.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 17 ==== osd_map(62..65 src has 1..65) v4 ==== 1820+0+0 (secure 0 0 0) 0x7f9fb8057570 con 0x56259b07cae0 2024-01-21T04:35:42.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(62..65 src has 1..65) v4 2024-01-21T04:35:42.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [62,65] > 61 2024-01-21T04:35:42.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 62 2024-01-21T04:35:42.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:42.586 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 62 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 63 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 63 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 64 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 64 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 65 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 65 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:42.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:42.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:38.890+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:42.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:39.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:42.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:39.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:42.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:39.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:42.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:40.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:42.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:40.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:42.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:40.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:42.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:41.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:42.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:41.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:42.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:41.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:42.696 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:42.688+0000 12a57640 -1 osd.3 66 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:42.877 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:42.734+0000 ea4f640 -1 osd.0 66 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4457.0:2 27.6 27:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:43.025 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:42.960+0000 ea4f640 -1 osd.1 66 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:43.879 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:42.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:43.879 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:42.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:43.879 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:42.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:43.879 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:43.879 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.838+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.838+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.838+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.838+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.838+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.838+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=66}) v3 -- 0x7f9f90003b80 con 0x56259b07cae0 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.838+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003ef0 con 0x56259b068c80 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 18 ==== osd_map(66..70 src has 1..70) v4 ==== 4133+0+0 (secure 0 0 0) 0x7f9fb8057570 con 0x56259b07cae0 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(66..70 src has 1..70) v4 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [66,70] > 65 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 66 2024-01-21T04:35:43.880 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 66 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 67 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 67 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 68 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 68 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 69 2024-01-21T04:35:43.881 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:43.928 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:43.866+0000 12a57640 -1 osd.3 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:43.963 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:43.895+0000 ea4f640 -1 osd.0 66 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4444.0:2 9.16 9.3fa85516 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:44.156 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:44.108+0000 ea4f640 -1 osd.1 67 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:44.364 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:35:44.313+0000 12a57640 -1 osd.4 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4673.0:2 64.1b 64.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-21T04:35:44.877 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Running main() from gmock_main.cc 2024-01-21T04:35:44.877 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-01-21T04:35:44.877 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [----------] Global test environment set-up. 2024-01-21T04:35:44.877 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-01-21T04:35:44.877 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-01-21T04:35:44.877 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: handle_notify cookie 93884015044016 notify_id 171798691841 notifier_gid 4469 2024-01-21T04:35:44.877 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (25412 ms) 2024-01-21T04:35:44.877 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-01-21T04:35:44.877 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Trying... 2024-01-21T04:35:44.877 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: handle_notify cookie 93884027674928 notify_id 261993005056 notifier_gid 4655 2024-01-21T04:35:44.877 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Waiting for 3.000000000s 2024-01-21T04:35:44.878 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Timed out. 2024-01-21T04:35:44.878 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Flushing... 2024-01-21T04:35:44.878 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: Flushed... 2024-01-21T04:35:44.878 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (35517 ms) 2024-01-21T04:35:44.878 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (60929 ms total) 2024-01-21T04:35:44.878 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: 2024-01-21T04:35:44.878 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [----------] Global test environment tear-down 2024-01-21T04:35:44.878 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (60929 ms total) 2024-01-21T04:35:44.878 INFO:tasks.workunit.client.0.smithi069.stdout: watch_notify: [ PASSED ] 2 tests. 2024-01-21T04:35:45.285 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:45.191+0000 ea4f640 -1 osd.0 68 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4444.0:2 9.16 9:68aa15fc:test-rados-api-smithi069-39960-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:45.334 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:45.314+0000 ea4f640 -1 osd.1 68 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:45.462 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:45.294+0000 12a57640 -1 osd.3 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:45.559 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:35:45.501+0000 12a57640 -1 osd.4 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4673.0:2 64.1b 64.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-21T04:35:45.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: Running main() from gmock_main.cc 2024-01-21T04:35:45.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-01-21T04:35:45.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] Global test environment set-up. 2024-01-21T04:35:45.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-01-21T04:35:45.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-01-21T04:35:45.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolList (6087 ms) 2024-01-21T04:35:45.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-01-21T04:35:45.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (5817 ms) 2024-01-21T04:35:45.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-01-21T04:35:45.992 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3922 ms) 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (5668 ms) 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (5793 ms) 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (9692 ms) 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (8381 ms) 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (7030 ms) 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (10054 ms) 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] 9 tests from LibRadosPools (62444 ms total) 2024-01-21T04:35:45.993 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: 2024-01-21T04:35:45.994 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [----------] Global test environment tear-down 2024-01-21T04:35:45.994 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (62444 ms total) 2024-01-21T04:35:45.994 INFO:tasks.workunit.client.0.smithi069.stdout: api_pool: [ PASSED ] 9 tests. 2024-01-21T04:35:46.306 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:46.211+0000 ea4f640 -1 osd.0 70 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4444.0:2 9.16 9:68aa15fc:test-rados-api-smithi069-39960-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:46.315 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:46.298+0000 12a57640 -1 osd.3 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:46.416 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:46.404+0000 ea4f640 -1 osd.1 69 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:46.584 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:35:46.552+0000 12a57640 -1 osd.4 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4673.0:2 64.1b 64.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-21T04:35:47.447 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:47.416+0000 ea4f640 -1 osd.1 70 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:47.538 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:47.532+0000 12a57640 -1 osd.3 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:47.823 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:35:47.811+0000 12a57640 -1 osd.4 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4673.0:2 64.1b 64.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-21T04:35:47.971 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:47.848+0000 ea4f640 -1 osd.0 72 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4444.0:2 9.16 9:68aa15fc:test-rados-api-smithi069-39960-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:48.757 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:48.582+0000 ea4f640 -1 osd.1 71 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:48.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 69 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:48.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:48.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 70 2024-01-21T04:35:48.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:48.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 70 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:48.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:48.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:48.893 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:43.878+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:44.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:44.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:44.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:45.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:45.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:45.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:46.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:46.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:46.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:46.585+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:47.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:47.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:47.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:48.894 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:48.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:48.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.839+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:48.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.839+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:48.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.839+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:48.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.839+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:48.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.839+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:48.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.839+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=71}) v3 -- 0x7f9f90003b80 con 0x56259b07cae0 2024-01-21T04:35:48.895 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.839+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003ef0 con 0x56259b068c80 2024-01-21T04:35:48.972 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:48.961+0000 ea4f640 -1 osd.0 72 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4444.0:2 9.16 9:68aa15fc:test-rados-api-smithi069-39960-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:49.031 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:35:48.888+0000 12a57640 -1 osd.4 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4673.0:2 64.1b 64.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-21T04:35:49.264 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:49.259+0000 12a57640 -1 osd.3 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:49.966 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:35:49.952+0000 12a57640 -1 osd.4 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4673.0:2 64.1b 64.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-21T04:35:50.100 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:50.024+0000 ea4f640 -1 osd.1 72 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:50.187 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:50.069+0000 ea4f640 -1 osd.0 73 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4444.0:2 9.16 9:68aa15fc:test-rados-api-smithi069-39960-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:50.306 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:50.243+0000 12a57640 -1 osd.3 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:51.077 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:51.071+0000 ea4f640 -1 osd.1 73 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:51.097 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:51.087+0000 ea4f640 -1 osd.0 74 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4444.0:2 9.16 9:68aa15fc:test-rados-api-smithi069-39960-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:51.126 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:35:51.102+0000 12a57640 -1 osd.4 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4673.0:2 64.1b 64.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-21T04:35:51.243 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 19 ==== osd_map(71..75 src has 1..75) v4 ==== 2759+0+0 (secu api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-01-21T04:35:51.243 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-01-21T04:35:51.244 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-01-21T04:35:51.244 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-01-21T04:35:51.244 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (67496 ms) 2024-01-21T04:35:51.244 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (67496 ms total) 2024-01-21T04:35:51.244 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: 2024-01-21T04:35:51.244 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-01-21T04:35:51.244 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (67496 ms total) 2024-01-21T04:35:51.244 INFO:tasks.workunit.client.0.smithi069.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-01-21T04:35:51.353 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:51.299+0000 12a57640 -1 osd.3 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout:re 0 0 0) 0x7f9fb8057570 con 0x56259b07cae0 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(71..75 src has 1..75) v4 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [71,75] > 70 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 71 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 71 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 72 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 72 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:51.587 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 73 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 73 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 74 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 74 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 75 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 75 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:48.892+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:51.588 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:49.586+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:51.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:49.586+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:51.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:49.586+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:51.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:50.586+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:51.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:50.586+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:51.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:50.586+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:51.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:51.586+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:52.261 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:52.208+0000 ea4f640 -1 osd.0 75 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4444.0:2 9.16 9:68aa15fc:test-rados-api-smithi069-39960-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:35:52.536 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:52.459+0000 ea4f640 -1 osd.1 74 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:52.716 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:52.643+0000 12a57640 -1 osd.3 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:52.837 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:20.744 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:52.905 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:20.812 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:52.914 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:03:20.789 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:52.961 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:03:20.836 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:52.973 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:20.880 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:53.025 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:20.932 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:53.134 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:03:21.010 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:53.192 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:21.099 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:53.215 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:21.122 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:35:53.805 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:53.789+0000 12a57640 -1 osd.3 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:51.586+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:51.586+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:52.585+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:52.585+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:52.585+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.586+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.586+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.586+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.839+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.839+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.839+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.839+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:53.853 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.839+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.839+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=76}) v3 -- 0x7f9f90003b80 con 0x56259b07cae0 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.839+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003ef0 con 0x56259b068c80 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 20 ==== osd_map(76..78 src has 1..78) v4 ==== 2985+0+0 (secure 0 0 0) 0x7f9fb8057570 con 0x56259b07cae0 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(76..78 src has 1..78) v4 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [76,78] > 75 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 76 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 76 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 77 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 77 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 78 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:53.854 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 78 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:53.855 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:35:54.146 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:54.027+0000 ea4f640 -1 osd.1 75 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:54.948 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:54.932+0000 12a57640 -1 osd.3 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:55.370 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:35:55.265+0000 12a57640 -1 osd.4 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4673.0:2 64.1b 64.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-21T04:35:56.520 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:35:56.275+0000 ea4f640 -1 osd.1 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4407.0:2 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-01-21T04:35:56.993 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:35:56.983+0000 12a57640 -1 osd.3 78 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4451.0:2 39.0s0 39.0 (undecoded) ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e40) 2024-01-21T04:35:58.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:35:58.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:53.852+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:35:58.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:54.586+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:58.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:54.586+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:58.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:54.586+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:58.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:55.586+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:58.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:55.586+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:58.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:55.586+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:58.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:56.586+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:58.965 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:56.586+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:56.586+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:56.586+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:57.586+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:57.586+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:57.586+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.586+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.840+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.840+0000 7f9fc6889640 2 client.4407.objecter tid 2 on osd.1 is laggy 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.840+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.840+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.840+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.840+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=79}) v3 -- 0x7f9f90003b00 con 0x56259b07cae0 2024-01-21T04:35:58.966 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.840+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003ef0 con 0x56259b068c80 2024-01-21T04:35:58.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.963+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 21 ==== osd_map(79..83 src has 1..83) v4 ==== 5194+0+0 (secure 0 0 0) 0x7f9fb8057570 con 0x56259b07cae0 2024-01-21T04:35:58.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.963+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(79..83 src has 1..83) v4 2024-01-21T04:35:58.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.963+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [79,83] > 78 2024-01-21T04:35:58.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.963+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 79 2024-01-21T04:35:58.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:35:58.967 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 79 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:35:59.040 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:58.901+0000 ea4f640 -1 osd.0 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4444.0:2 9.16 9:68aa15fc:test-rados-api-smithi069-39960-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:36:00.007 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:35:59.939+0000 ea4f640 -1 osd.0 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4444.0:2 9.16 9:68aa15fc:test-rados-api-smithi069-39960-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:36:00.629 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:36:00.539+0000 12a57640 -1 osd.4 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4673.0:2 64.1b 64:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-21T04:36:00.933 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:36:00.924+0000 ea4f640 -1 osd.0 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4444.0:2 9.16 9:68aa15fc:test-rados-api-smithi069-39960-2::foo:head [call lock.lock in=57b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-01-21T04:36:01.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:01.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 80 2024-01-21T04:36:01.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:36:01.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 80 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:01.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:01.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 81 2024-01-21T04:36:01.256 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 81 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 82 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 82 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 83 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 10 client.4407.objecter checking op 2 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 83 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:58.964+0000 7f9fbd7fa640 20 client.4407.objecter 2 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:59.587+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:01.257 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:59.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:35:59.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:00.587+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:00.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:00.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.254+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 2 ==== osd_op_reply(2 exist [writefull 0~5] v52'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.254+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(2 exist [writefull 0~5] v52'1 uv1 ondisk = 0) v8 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 2 ondisk uv 1 in 12.1 attempt 0 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 2 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 2 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 2 2024-01-21T04:36:01.258 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 10 librados: Objecter returned from writefull r=0 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b12af10 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 83 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 3 osd.1 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 3 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 15 client.4407.objecter _send_op 3 to 12.1 on osd.1 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:3 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) v8 -- 0x56259b063c50 con 0x56259b068c80 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:01.259 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:01.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b063fb0 2024-01-21T04:36:01.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 83 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:01.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:01.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:01.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:01.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 4 osd.1 2024-01-21T04:36:01.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:01.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 4 2024-01-21T04:36:01.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 15 client.4407.objecter _send_op 4 to 12.1 on osd.1 2024-01-21T04:36:01.260 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:4 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) v8 -- 0x56259b064450 con 0x56259b068c80 2024-01-21T04:36:02.110 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T04:36:02.039+0000 12a57640 -1 osd.4 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4673.0:2 64.1b 64:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-01-21T04:36:06.935 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:06.707+0000 12a57640 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.144:6820 osd.6 since back 2024-01-21T04:34:45.340896+0000 front 2024-01-21T04:36:04.678583+0000 (oldest deadline 2024-01-21T04:36:06.437631+0000) 2024-01-21T04:36:07.907 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:07.866+0000 12a57640 -1 osd.3 87 heartbeat_check: no reply from 172.21.15.144:6820 osd.6 since back 2024-01-21T04:34:45.340896+0000 front 2024-01-21T04:36:04.678583+0000 (oldest deadline 2024-01-21T04:36:06.437631+0000) 2024-01-21T04:36:10.378 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:36:10.194+0000 ea4f640 -1 osd.1 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4691.0:2 71.1e 71.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e61) 2024-01-21T04:36:11.313 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:36:11.291+0000 ea4f640 -1 osd.1 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4691.0:2 71.1e 71.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e61) 2024-01-21T04:36:12.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:12.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.255+0000 7f9fc7239780 5 client.4407.objecter 2 in flight 2024-01-21T04:36:12.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.587+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:12.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:12.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:01.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:12.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:02.587+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:12.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:02.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:12.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:02.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:12.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:03.587+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:12.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:03.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:03.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:03.841+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:04.587+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:04.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:04.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:05.587+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:05.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:05.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:06.587+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:06.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:06.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:06.587+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:36:12.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:07.587+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:07.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:07.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:08.587+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:08.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:08.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:08.842+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:09.587+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:09.587+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:09.587+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:10.588+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:10.588+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:10.588+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:11.588+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:12.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:11.588+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:12.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:11.588+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:13.440 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:41.347 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:13.441 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:03:41.316 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:13.443 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:41.350 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:13.450 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:03:41.325 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:13.505 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:41.412 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:13.506 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:41.413 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:13.608 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:03:41.482 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:13.663 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:03:41.570 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:13.763 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:03:41.670 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:13.939 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:12.588+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:12.588+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:12.588+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.588+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.588+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.588+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.843+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.843+0000 7f9fc6889640 2 client.4407.objecter tid 3 on osd.1 is laggy 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.843+0000 7f9fc6889640 2 client.4407.objecter tid 4 on osd.1 is laggy 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.843+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.843+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.843+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:36:13.940 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.843+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=84}) v3 -- 0x7f9f90003aa0 con 0x56259b07cae0 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.843+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003ef0 con 0x56259b068c80 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.937+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 22 ==== osd_map(84..96 src has 1..96) v4 ==== 9931+0+0 (secure 0 0 0) 0x7f9fb80577e0 con 0x56259b07cae0 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.937+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(84..96 src has 1..96) v4 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [84,96] > 83 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 84 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 84 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 84 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 85 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:13.941 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 85 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:13.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:13.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 85 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:13.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 86 2024-01-21T04:36:13.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:13.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 86 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:13.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:13.942 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 86 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 87 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 87 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 87 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 88 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 88 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:13.943 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 88 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 89 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 89 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 89 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 90 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:13.944 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 90 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 90 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 91 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 91 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 91 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 92 2024-01-21T04:36:13.945 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 92 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 92 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 93 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 93 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 93 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 94 2024-01-21T04:36:13.946 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:16.160 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:36:16.126+0000 ea4f640 -1 osd.1 94 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4691.0:2 71.1e 71.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e61) 2024-01-21T04:36:17.248 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:36:17.234+0000 ea4f640 -1 osd.1 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4691.0:2 71.1e 71.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e61) 2024-01-21T04:36:17.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 94 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:17.589 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 94 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 95 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 95 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 95 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 96 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:17.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 96 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 96 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter 3 12.d3a59201 osd.1 exist [read 0~256] 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:13.938+0000 7f9fbd7fa640 20 client.4407.objecter 4 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:14.588+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:14.588+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:14.588+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:15.588+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:15.588+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:15.588+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:17.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:16.588+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:17.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:16.588+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:17.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:16.588+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:17.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:16.588+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:17.588+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:17.588+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:17.588+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.588+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.588+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.588+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.844+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.844+0000 7f9fc6889640 2 client.4407.objecter tid 3 on osd.1 is laggy 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.844+0000 7f9fc6889640 2 client.4407.objecter tid 4 on osd.1 is laggy 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.844+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.844+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:36:18.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.844+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.844+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=97}) v3 -- 0x7f9f90003aa0 con 0x56259b07cae0 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.844+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003ef0 con 0x56259b068c80 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.846+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 23 ==== osd_map(97..100 src has 1..100) v4 ==== 1076+0+0 (secure 0 0 0) 0x7f9fb80577e0 con 0x56259b07cae0 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.846+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(97..100 src has 1..100) v4 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.846+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [97,100] > 96 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.846+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 97 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.846+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.846+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 97 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.846+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 97 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 98 2024-01-21T04:36:18.849 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:18.850 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 98 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:18.850 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:18.850 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:18.850 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 98 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:21.071 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:36:20.926+0000 ea4f640 -1 osd.1 99 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4691.0:2 71.1e 71.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e61) 2024-01-21T04:36:22.204 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:36:22.103+0000 ea4f640 -1 osd.1 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4691.0:2 71.1e 71.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e61) 2024-01-21T04:36:23.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:23.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 99 2024-01-21T04:36:23.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:23.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 99 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:23.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:23.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:23.846 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 99 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 100 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 100 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 100 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter 3 12.d3a59201 osd.1 exist [read 0~256] 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:18.847+0000 7f9fbd7fa640 20 client.4407.objecter 4 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:19.588+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:19.588+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:23.847 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:19.588+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:20.588+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:20.588+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:20.588+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:21.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:21.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:21.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:22.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:22.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:22.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:23.848 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:23.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.845+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:36:23.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.845+0000 7f9fc6889640 2 client.4407.objecter tid 3 on osd.1 is laggy 2024-01-21T04:36:23.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.845+0000 7f9fc6889640 2 client.4407.objecter tid 4 on osd.1 is laggy 2024-01-21T04:36:23.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.845+0000 7f9fc6889640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T04:36:23.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.845+0000 7f9fc6889640 10 monclient: _renew_subs 2024-01-21T04:36:23.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.845+0000 7f9fc6889640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T04:36:23.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.845+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=101}) v3 -- 0x7f9f90003aa0 con 0x56259b07cae0 2024-01-21T04:36:23.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.845+0000 7f9fc6889640 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- ping magic: 0 v1 -- 0x7f9f90003ef0 con 0x56259b068c80 2024-01-21T04:36:23.865 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== mon.0 v2:172.21.15.69:3300/0 24 ==== osd_map(101..105 src has 1..105) v4 ==== 4806+0+0 (secure 0 0 0) 0x7f9fb80577e0 con 0x56259b07cae0 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(101..105 src has 1..105) v4 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [101,105] > 100 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 101 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 101 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 101 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 102 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 102 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:23.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:23.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 102 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:23.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:23.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 103 2024-01-21T04:36:23.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:23.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 103 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:26.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:26.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:26.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 103 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:26.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:26.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 104 2024-01-21T04:36:26.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:26.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 104 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 104 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 105 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 10 client.4407.objecter checking op 3 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 105 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 10 client.4407.objecter checking op 4 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 105 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter 3 12.d3a59201 osd.1 exist [read 0~256] 2024-01-21T04:36:26.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:23.864+0000 7f9fbd7fa640 20 client.4407.objecter 4 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-01-21T04:36:26.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:24.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:26.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:24.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:26.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:24.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:26.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:25.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:26.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:25.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:26.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:25.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:26.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:26.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:26.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:26.595 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.589+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 3 ==== osd_map(84..100 src has 1..102) v4 ==== 10971+0+0 (crc 0 0 0) 0x7f9fb406d040 con 0x56259b068c80 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(84..100 src has 1..102) v4 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map ignoring epochs [84,100] <= 105 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fbd7fa640 20 client.4407.objecter 3 12.d3a59201 osd.1 exist [read 0~256] 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fbd7fa640 20 client.4407.objecter 4 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (25337 ms) 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 3 ondisk uv 1 in 12.1 attempt 0 2024-01-21T04:36:26.596 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 5 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 3 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 3 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 3 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 5 client.4407.objecter 1 in flight 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 6 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 4 ondisk uv 0 in 12.1 attempt 0 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 4 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 4 2024-01-21T04:36:26.597 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 4 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 5 ==== osd_map(101..101 src has 1..102) v4 ==== 546+0+0 (crc 0 0 0) 0x7f9fb405c8f0 con 0x56259b068c80 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(101..101 src has 1..102) v4 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map ignoring epochs [101,101] <= 105 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b063fb0 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 105 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 5 osd.1 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:26.598 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 5 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 15 client.4407.objecter _send_op 5 to 12.1 on osd.1 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:5 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e105) v8 -- 0x56259b12af10 con 0x56259b068c80 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b0647b0 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 105 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 6 osd.1 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 6 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 15 client.4407.objecter _send_op 6 to 12.1 on osd.1 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:6 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e105) v8 -- 0x56259b08b180 con 0x56259b068c80 2024-01-21T04:36:26.599 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:28.830 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:36:28.808+0000 ea4f640 -1 osd.0 108 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:9 18.f 18:f2f219cc:::7.obj:head [write 0~222 in=222b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79) 2024-01-21T04:36:29.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:26.592+0000 7f9fc7239780 5 client.4407.objecter 2 in flight 2024-01-21T04:36:29.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:27.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:29.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:27.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:29.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:27.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:29.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:28.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:29.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:28.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:29.019 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:28.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:28.846+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.017+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 7 ==== osd_map(106..106 src has 1..107) v4 ==== 1340+0+0 (crc 0 0 0) 0x7f9fb4068450 con 0x56259b068c80 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.017+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(106..106 src has 1..107) v4 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.017+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [106,106] > 105 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.017+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 106 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fbd7fa640 10 client.4407.objecter checking op 5 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 106 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fbd7fa640 10 client.4407.objecter checking op 6 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 106 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:29.020 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fbd7fa640 20 client.4407.objecter 5 12.d3a59201 osd.1 exist [read 0~256] 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fbd7fa640 20 client.4407.objecter 6 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.017+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 8 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 5 ondisk uv 1 in 12.1 attempt 0 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 5 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 5 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 5 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 5 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 5 client.4407.objecter 1 in flight 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 9 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:29.021 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 6 ondisk uv 0 in 12.1 attempt 0 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 6 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 6 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (2426 ms) 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 6 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b063fb0 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 106 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:29.022 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [read 0~256] tid 7 osd.1 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 7 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 15 client.4407.objecter _send_op 7 to 12.1 on osd.1 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:7 12.1 12:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x56259b0647b0 con 0x56259b068c80 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08bda0 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 106 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:29.023 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:29.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:29.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:29.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid noexist '@12' '@12' [read 0~256] tid 8 osd.1 2024-01-21T04:36:29.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:29.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 8 2024-01-21T04:36:29.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 15 client.4407.objecter _send_op 8 to 12.1 on osd.1 2024-01-21T04:36:29.287 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:8 12.1 12:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106) v8 -- 0x56259b08c240 con 0x56259b068c80 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.018+0000 7f9fc7239780 5 client.4407.objecter 2 in flight 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 11 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 10 ==== osd_map(107..107 src has 1..109) v4 ==== 320+0+0 (crc 0 0 0) 0x7f9fb406d020 con 0x56259b068c80 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(107..107 src has 1..109) v4 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 7 ondisk uv 1 in 12.1 attempt 0 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 5 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 7 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 7 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:29.288 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 7 2024-01-21T04:36:29.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 5 client.4407.objecter 1 in flight 2024-01-21T04:36:29.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [107,107] > 106 2024-01-21T04:36:29.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 107 2024-01-21T04:36:29.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 12 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:29.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T04:36:29.289 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:29.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fbd7fa640 10 client.4407.objecter checking op 8 2024-01-21T04:36:29.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 107 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:29.590 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fbd7fa640 20 client.4407.objecter 8 12.a7bbaa1 osd.1 noexist [read 0~256] 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 8 ondisk uv 0 in 12.1 attempt 0 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 8 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 8 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 8 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (268 ms) 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 20 librados: queue_aio_write 0x56259af65520 completion 0x56259af04170 write_seq 1 2024-01-21T04:36:29.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08bda0 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 107 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 9 osd.1 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 9 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 15 client.4407.objecter _send_op 9 to 12.1 on osd.1 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:9 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) v8 -- 0x56259b063fb0 con 0x56259b068c80 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:29.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.286+0000 7f9fc7239780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-01-21T04:36:29.820 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:36:29.811+0000 ea4f640 -1 osd.0 108 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:9 18.f 18:f2f219cc:::7.obj:head [write 0~222 in=222b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79) 2024-01-21T04:36:30.416 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.589+0000 7f9fa7ff handler_error: Running main() from gmock_main.cc 2024-01-21T04:36:30.416 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-01-21T04:36:30.416 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [----------] Global test environment set-up. 2024-01-21T04:36:30.417 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-01-21T04:36:30.417 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-01-21T04:36:30.417 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (106593 ms) 2024-01-21T04:36:30.417 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [----------] 1 test from neocls_handler_error (106593 ms total) 2024-01-21T04:36:30.417 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: 2024-01-21T04:36:30.417 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [----------] Global test environment tear-down 2024-01-21T04:36:30.417 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [==========] 1 test from 1 test suite ran. (106593 ms total) 2024-01-21T04:36:30.417 INFO:tasks.workunit.client.0.smithi069.stdout: handler_error: [ PASSED ] 1 test. 2024-01-21T04:36:30.832 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:36:30.827+0000 ea4f640 -1 osd.0 109 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:9 18.f 18:f2f219cc:::7.obj:head [write 0~222 in=222b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79) 2024-01-21T04:36:31.890 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:36:31.884+0000 ea4f640 -1 osd.0 110 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4424.0:9 18.f 18:f2f219cc:::7.obj:head [write 0~222 in=222b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79) 2024-01-21T04:36:32.119 INFO:tasks.workunit.client.0.smithi069.stdout:f640 10 monclient: tick 2024-01-21T04:36:32.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:32.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:29.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:32.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:30.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:32.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:30.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:32.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:30.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:32.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:31.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:32.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:31.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:32.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:31.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:32.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.117+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 13 ==== osd_op_reply(9 exist [write 0~5] v107'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:32.119 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.117+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(9 exist [write 0~5] v107'2 uv2 ondisk = 0) v8 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.117+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.117+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 9 ondisk uv 2 in 12.1 attempt 0 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.117+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.117+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 9 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.117+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 9 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.117+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.117+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 9 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.117+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.117+0000 7f9fc4e26640 20 librados: complete_aio_write 0x56259af04170 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (2832 ms) 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-01-21T04:36:32.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 20 librados: queue_aio_write 0x56259af65520 completion 0x56259af0ca90 write_seq 2 2024-01-21T04:36:32.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08bda0 2024-01-21T04:36:32.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 107 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:32.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:32.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:32.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:32.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 10 osd.1 2024-01-21T04:36:32.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:32.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 10 2024-01-21T04:36:32.983 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:36:32.952+0000 ea4f640 -1 osd.0 111 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4424.0:9 18.f 18:f2f219cc:::7.obj:head [write 0~222 in=222b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79) 2024-01-21T04:36:33.250 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:01.156 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:33.254 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:01.161 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:33.278 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:04:01.153 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:33.282 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:04:01.157 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:33.324 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:01.231 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:33.330 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:01.237 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:33.381 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:04:01.256 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:33.412 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:01.319 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:33.457 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:01.364 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:36:34.048 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:36:34.040+0000 ea4f640 -1 osd.0 112 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4424.0:9 18.f 18:f2f219cc:::7.obj:head [write 0~222 in=222b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79) 2024-01-21T04:36:34.253 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 15 client.4407.objecter _send_op 10 to 12.1 on osd.1 2024-01-21T04:36:34.253 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:10 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) v8 -- 0x56259b063fb0 con 0x56259b068c80 2024-01-21T04:36:34.253 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:34.253 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:34.253 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.118+0000 7f9fc7239780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-01-21T04:36:34.253 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 14 ==== osd_map(108..110 src has 1..111) v4 ==== 840+0+0 (crc 0 0 0) 0x7f9fb406d040 con 0x56259b068c80 2024-01-21T04:36:34.253 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(108..110 src has 1..111) v4 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [108,110] > 107 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 108 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 10 client.4407.objecter checking op 10 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 108 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 109 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 10 client.4407.objecter checking op 10 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 109 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 110 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 10 client.4407.objecter checking op 10 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 110 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:34.254 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.185+0000 7f9fbd7fa640 20 client.4407.objecter 10 12.d3a59201 osd.1 exist [write 0~5 in=5b] 2024-01-21T04:36:34.255 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:34.255 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:34.255 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:32.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:34.255 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:33.590+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:34.255 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:33.590+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:34.255 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:33.590+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:34.255 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:33.848+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:36:34.313 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 15 ==== osd_op_reply(10 exist [write 0~5] v110'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:34.313 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(10 exist [write 0~5] v110'3 uv3 ondisk = 0) v8 2024-01-21T04:36:34.313 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:34.313 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 10 ondisk uv 3 in 12.1 attempt 0 2024-01-21T04:36:34.313 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:34.313 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 10 2024-01-21T04:36:34.313 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 10 2024-01-21T04:36:34.313 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:34.313 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 10 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc4e26640 20 librados: complete_aio_write 0x56259af0ca90 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (2134 ms) 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 20 librados: queue_aio_write 0x56259af65520 completion 0x56259b064190 write_seq 3 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08bda0 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 110 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [write 0~5 in=5b] tid 11 osd.1 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:34.314 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 11 2024-01-21T04:36:34.315 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 15 client.4407.objecter _send_op 11 to 12.1 on osd.1 2024-01-21T04:36:34.315 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:11 12.1 12:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) v8 -- 0x56259b08f780 con 0x56259b068c80 2024-01-21T04:36:34.315 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:34.315 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:34.315 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.252+0000 7f9fc7239780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-01-21T04:36:34.315 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 16 ==== osd_map(111..112 src has 1..112) v4 ==== 914+0+0 (crc 0 0 0) 0x7f9fb405c8d0 con 0x56259b068c80 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(111..112 src has 1..112) v4 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [111,112] > 110 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 111 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 10 client.4407.objecter checking op 11 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 111 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 112 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 10 client.4407.objecter checking op 11 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 112 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.312+0000 7f9fbd7fa640 20 client.4407.objecter 11 12.d3a59201 osd.1 exist [write 0~5 in=5b] 2024-01-21T04:36:35.728 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.590+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.590+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:34.590+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.590+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.590+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.590+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.726+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 17 ==== osd_op_reply(11 exist [write 0~5] v112'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.726+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(11 exist [write 0~5] v112'4 uv4 ondisk = 0) v8 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.726+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.726+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 11 ondisk uv 4 in 12.1 attempt 0 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.726+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.726+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 11 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.726+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 11 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.726+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.726+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 11 2024-01-21T04:36:35.729 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.726+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:35.730 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.726+0000 7f9fc4e26640 20 librados: complete_aio_write 0x56259b064190 2024-01-21T04:36:35.730 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1475 ms) 2024-01-21T04:36:35.730 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-01-21T04:36:35.730 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08bda0 2024-01-21T04:36:35.752 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 112 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 12 osd.1 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 12 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 15 client.4407.objecter _send_op 12 to 12.1 on osd.1 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:12 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112) v8 -- 0x56259b08f110 con 0x56259b068c80 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08f470 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 112 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:35.753 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 13 osd.1 2024-01-21T04:36:35.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 13 2024-01-21T04:36:35.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 15 client.4407.objecter _send_op 13 to 12.1 on osd.1 2024-01-21T04:36:35.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:13 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112) v8 -- 0x56259b08c5a0 con 0x56259b068c80 2024-01-21T04:36:35.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:35.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.727+0000 7f9fc7239780 5 client.4407.objecter 2 in flight 2024-01-21T04:36:35.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 18 ==== osd_map(113..113 src has 1..113) v4 ==== 296+0+0 (crc 0 0 0) 0x7f9fb406d040 con 0x56259b068c80 2024-01-21T04:36:35.754 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(113..113 src has 1..113) v4 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 19 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [113,113] > 112 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 113 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 10 client.4407.objecter checking op 12 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 113 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 10 client.4407.objecter checking op 13 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 113 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:35.758 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 20 client.4407.objecter 12 12.d3a59201 osd.1 exist [read 0~0] 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fbd7fa640 20 client.4407.objecter 13 12.a7bbaa1 osd.1 noexist [read 0~0] 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 12 ondisk uv 4 in 12.1 attempt 0 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 5 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 12 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 12 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 12 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.751+0000 7f9fc4e26640 5 client.4407.objecter 1 in flight 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 20 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 13 ondisk uv 0 in 12.1 attempt 0 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc4e26640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:35.759 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 13 2024-01-21T04:36:35.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 13 2024-01-21T04:36:35.760 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 13 2024-01-21T04:36:35.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:35.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (30 ms) 2024-01-21T04:36:35.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-01-21T04:36:35.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08f110 2024-01-21T04:36:35.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 113 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:35.778 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 14 osd.1 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 14 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 15 client.4407.objecter _send_op 14 to 12.1 on osd.1 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:14 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) v8 -- 0x56259b08beb0 con 0x56259b068c80 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08f6c0 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 113 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:35.779 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 15 osd.1 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 15 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 15 client.4407.objecter _send_op 15 to 12.1 on osd.1 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:15 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) v8 -- 0x56259b08c5a0 con 0x56259b068c80 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.756+0000 7f9fc7239780 5 client.4407.objecter 2 in flight 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 21 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 14 ondisk uv 4 in 12.1 attempt 0 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 5 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 14 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 14 2024-01-21T04:36:35.780 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 14 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 5 client.4407.objecter 1 in flight 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 22 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 15 ondisk uv 0 in 12.1 attempt 0 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 15 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 15 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 15 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (20 ms) 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-01-21T04:36:35.781 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08f6c0 2024-01-21T04:36:35.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 113 base exist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:35.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:35.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:35.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:35.782 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [read 0~0] tid 16 osd.1 2024-01-21T04:36:35.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:35.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 16 2024-01-21T04:36:35.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 15 client.4407.objecter _send_op 16 to 12.1 on osd.1 2024-01-21T04:36:35.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:16 12.1 12:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) v8 -- 0x56259b08f110 con 0x56259b068c80 2024-01-21T04:36:35.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.812 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08cd70 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 113 base noexist @12 precalc_pgid 0 pgid 0.0 is_read 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 20 client.4407.objecter _calc_target target noexist @12 -> pgid 12.a7bbaa1 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.a7bbaa1 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid noexist '@12' '@12' [read 0~0] tid 17 osd.1 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 17 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 15 client.4407.objecter _send_op 17 to 12.1 on osd.1 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:17 12.1 12:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) v8 -- 0x56259b08d210 con 0x56259b068c80 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.776+0000 7f9fc7239780 5 client.4407.objecter 2 in flight 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.810+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 23 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.810+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-01-21T04:36:35.813 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.810+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:35.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.810+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 16 ondisk uv 4 in 12.1 attempt 0 2024-01-21T04:36:35.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.810+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 5 2024-01-21T04:36:35.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.810+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 16 2024-01-21T04:36:35.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.810+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 16 2024-01-21T04:36:35.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.810+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 5 2024-01-21T04:36:35.814 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.810+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 16 2024-01-21T04:36:36.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.810+0000 7f9fc4e26640 5 client.4407.objecter 1 in flight 2024-01-21T04:36:36.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 24 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:36.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-01-21T04:36:36.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:36.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 17 ondisk uv 0 in 12.1 attempt 0 2024-01-21T04:36:36.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc4e26640 10 client.4407.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-01-21T04:36:36.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 17 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 17 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 17 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (38 ms) 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 20 librados: queue_aio_write 0x56259af65520 completion 0x56259b08f5b0 write_seq 4 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08f6c0 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 113 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 18 osd.1 2024-01-21T04:36:36.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:36.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 18 2024-01-21T04:36:36.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 15 client.4407.objecter _send_op 18 to 12.1 on osd.1 2024-01-21T04:36:36.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:18 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) v8 -- 0x56259b08f110 con 0x56259b068c80 2024-01-21T04:36:36.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:36.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:35.814+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:36.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.589+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:36.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.589+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.589+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.589+0000 7f9fa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 25 ==== osd_op_reply(18 exist [writefull 0~5] v113'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9fb404abc0 con 0x56259b068c80 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(18 exist [writefull 0~5] v113'5 uv5 ondisk = 0) v8 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 18 ondisk uv 5 in 12.1 attempt 0 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 18 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 18 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 18 2024-01-21T04:36:37.145 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc4e26640 20 librados: complete_aio_write 0x56259b08f5b0 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1100 ms) 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 20 librados: queue_aio_write 0x56259af65520 completion 0x56259b08fcd0 write_seq 5 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08f6c0 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 113 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 19 osd.1 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 19 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 15 client.4407.objecter _send_op 19 to 12.1 on osd.1 2024-01-21T04:36:37.146 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:19 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) v8 -- 0x56259b08c5a0 con 0x56259b068c80 2024-01-21T04:36:37.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:37.147 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:36.915+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:37.290 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:36:37.279+0000 ea4f640 -1 osd.0 114 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4712.0:2 76.15 76.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e85) 2024-01-21T04:36:38.984 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:36:38.945+0000 ea4f640 -1 osd.0 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4712.0:2 76.15 76:ad551702:::test:head [create,setxattr key (5) in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) 2024-01-21T04:36:39.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.143+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 26 ==== osd_map(114..114 src has 1..114) v4 ==== 1048+0+0 (crc 0 0 0) 0x7f9fb406d040 con 0x56259b068c80 2024-01-21T04:36:39.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.143+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(114..114 src has 1..114) v4 2024-01-21T04:36:39.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.143+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [114,114] > 113 2024-01-21T04:36:39.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.143+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 114 2024-01-21T04:36:39.297 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.143+0000 7f9fbd7fa640 10 client.4407.objecter checking op 19 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.143+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 114 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.143+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.143+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.143+0000 7f9fbd7fa640 20 client.4407.objecter 19 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.590+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.590+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:37.590+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:38.590+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:38.590+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:38.590+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:38.848+0000 7f9fc6889640 10 client.4407.objecter tick 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 27 ==== osd_op_reply(19 exist [writefull 0~5] v114'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9fb407b040 con 0x56259b068c80 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(19 exist [writefull 0~5] v114'6 uv6 ondisk = 0) v8 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:39.298 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 19 ondisk uv 6 in 12.1 attempt 0 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 19 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 19 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 19 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc4e26640 20 librados: complete_aio_write 0x56259b08fcd0 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (2381 ms) 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 20 librados: queue_aio_write 0x56259af65520 completion 0x56259b08ca50 write_seq 6 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 10 client.4407.objecter _op_submit op 0x56259b08f6c0 2024-01-21T04:36:39.299 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 20 client.4407.objecter _calc_target epoch 114 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:40.043 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T04:36:40.038+0000 ea4f640 -1 osd.0 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4712.0:2 76.15 76:ad551702:::test:head [create,setxattr key (5) in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85) 2024-01-21T04:36:41.591 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 10 client.4407.objecter _calc_target raw pgid 12.d3a59201 -> actual 12.1 acting [1,6] primary 1 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 20 client.4407.objecter _get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 10 client.4407.objecter _op_submit oid exist '@12' '@12' [writefull 0~5 in=5b] tid 20 osd.1 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 20 client.4407.objecter get_session s=0x56259af29570 osd=1 3 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 15 client.4407.objecter _session_op_assign 1 20 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 15 client.4407.objecter _send_op 20 to 12.1 on osd.1 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- osd_op(unknown.0.0:20 12.1 12:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) v8 -- 0x56259b08c5a0 con 0x56259b068c80 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.296+0000 7f9fc7239780 5 client.4407.objecter 1 in flight 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 28 ==== osd_map(115..116 src has 1..118) v4 ==== 2042+0+0 (crc 0 0 0) 0x7f9fb4073e10 con 0x56259b068c80 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_map(115..116 src has 1..118) v4 2024-01-21T04:36:41.592 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map got epochs [115,116] > 114 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 115 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 10 client.4407.objecter checking op 20 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 115 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 3 client.4407.objecter handle_osd_map decoding incremental epoch 116 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 10 client.4407.objecter checking op 20 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target epoch 116 base exist @12 precalc_pgid 0 pgid 0.0 is_write 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 20 client.4407.objecter _calc_target target exist @12 -> pgid 12.d3a59201 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.552+0000 7f9fbd7fa640 20 client.4407.objecter 20 12.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.590+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.590+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:39.590+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:40.590+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:41.593 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:40.590+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:41.594 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:40.590+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:42.367 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:42.305+0000 12a57640 -1 osd.3 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:36:42.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:41.590+0000 7f9fa7fff640 10 monclient: tick 2024-01-21T04:36:42.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:41.590+0000 7f9fa7fff640 10 monclient: _check_auth_tickets 2024-01-21T04:36:42.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:41.590+0000 7f9fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T04:36:42.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.523+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 <== osd.1 v2:172.21.15.69:6808/3868480563 29 ==== osd_op_reply(20 exist [writefull 0~5] v116'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f9fb407b040 con 0x56259b068c80 2024-01-21T04:36:42.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.523+0000 7f9fc4e26640 10 client.4407.objecter ms_dispatch 0x56259af5eb50 osd_op_reply(20 exist [writefull 0~5] v116'7 uv7 ondisk = 0) v8 2024-01-21T04:36:42.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.523+0000 7f9fc4e26640 10 client.4407.objecter in handle_osd_op_reply 2024-01-21T04:36:42.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc4e26640 7 client.4407.objecter handle_osd_op_reply 20 ondisk uv 7 in 12.1 attempt 0 2024-01-21T04:36:42.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc4e26640 10 client.4407.objecter op 0 rval 0 len 0 2024-01-21T04:36:42.525 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc4e26640 15 client.4407.objecter handle_osd_op_reply completed tid 20 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc4e26640 15 client.4407.objecter _finish_op 20 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc4e26640 20 client.4407.objecter put_session s=0x56259af29570 osd=1 4 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc4e26640 15 client.4407.objecter _session_op_remove 1 20 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc4e26640 5 client.4407.objecter 0 in flight 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc4e26640 20 librados: complete_aio_write 0x56259b08ca50 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (3228 ms) 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 10 librados: watch_flush enter 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 10 librados: watch_flush exit 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 10 client.4407.objecter close_session for osd.1 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x56259b068c80 msgr2=0x56259b12d8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x56259b068c80 0x56259b12d8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 20 client.4407.objecter put_session s=0x56259af29570 osd=1 1 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 20 client.4407.objecter shutdown clearing up homeless session... 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 10 client.4407.objecter successfully canceled tick 2024-01-21T04:36:42.526 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f9fac06aad0 msgr2=0x7f9fac06cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:36:42.527 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f9fac06aad0 0x7f9fac06cf90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9fb0005f80 tx=0x7f9fb0005530 comp rx=0 tx=0).stop 2024-01-21T04:36:42.527 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 10 monclient: shutdown 2024-01-21T04:36:42.527 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T04:36:42.528 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56259b07cae0 msgr2=0x56259b069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T04:36:42.528 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 1 --2- 172.21.15.69:0/1833251071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56259b07cae0 0x56259b069650 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9fb800d4d0 tx=0x7f9fb800d990 comp rx=0 tx=0).stop 2024-01-21T04:36:42.528 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc4e26640 1 -- 172.21.15.69:0/1833251071 reap_dead start 2024-01-21T04:36:42.528 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 shutdown_connections 2024-01-21T04:36:42.528 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 >> 172.21.15.69:0/1833251071 conn(0x56259afedc70 msgr2=0x56259afec940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T04:36:42.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 shutdown_connections 2024-01-21T04:36:42.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 1 -- 172.21.15.69:0/1833251071 wait complete. 2024-01-21T04:36:42.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.524+0000 7f9fc7239780 1 librados: shutdown 2024-01-21T04:36:42.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] 12 tests from AsioRados (41269 ms total) 2024-01-21T04:36:42.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: 2024-01-21T04:36:42.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [----------] Global test environment tear-down 2024-01-21T04:36:42.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (118985 ms total) 2024-01-21T04:36:42.529 INFO:tasks.workunit.client.0.smithi069.stdout: api_asio: [ PASSED ] 12 tests. 2024-01-21T04:36:43.420 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:43.411+0000 12a57640 -1 osd.3 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:36:50.265 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:50.253+0000 12a57640 -1 osd.3 128 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:36:51.563 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:51.535+0000 12a57640 -1 osd.3 128 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:36:53.487 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:53.359+0000 12a57640 -1 osd.3 130 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:36:54.567 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:54.536+0000 12a57640 -1 osd.3 131 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:36:55.792 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:55.778+0000 12a57640 -1 osd.3 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:36:56.869 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:56.864+0000 12a57640 -1 osd.3 133 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:36:58.154 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:58.083+0000 12a57640 -1 osd.3 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:36:59.333 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:36:59.309+0000 12a57640 -1 osd.3 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:03.031 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:02.977+0000 12a57640 -1 osd.6 139 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4457.0:32 27.17 27:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) 2024-01-21T04:37:05.125 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:05.104+0000 12a57640 -1 osd.6 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4457.0:32 27.17 27:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) 2024-01-21T04:37:06.207 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:06.188+0000 12a57640 -1 osd.6 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4457.0:32 27.17 27:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) 2024-01-21T04:37:07.525 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:07.375+0000 12a57640 -1 osd.6 144 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4457.0:32 27.17 27:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) 2024-01-21T04:37:08.940 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:08.871+0000 12a57640 -1 osd.6 145 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4457.0:32 27.17 27:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) 2024-01-21T04:37:10.066 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:10.062+0000 12a57640 -1 osd.6 146 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4457.0:32 27.17 27:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) 2024-01-21T04:37:10.743 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:10.666+0000 12a57640 -1 osd.3 146 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:11.145 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:11.136+0000 12a57640 -1 osd.6 147 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4457.0:32 27.17 27:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) 2024-01-21T04:37:12.004 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:11.870+0000 12a57640 -1 osd.3 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:12.197 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:12.189+0000 12a57640 -1 osd.6 149 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4457.0:32 27.17 27:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) 2024-01-21T04:37:12.937 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:12.906+0000 12a57640 -1 osd.3 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:13.231 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:13.214+0000 12a57640 -1 osd.6 149 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4457.0:32 27.17 27:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) 2024-01-21T04:37:13.904 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:13.884+0000 12a57640 -1 osd.3 149 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:14.006 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:41.913 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:14.075 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:41.981 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:14.102 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:42.009 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:14.142 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:42.049 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:14.145 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:04:42.020 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:14.151 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:04:42.026 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:14.279 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:14.260+0000 12a57640 -1 osd.6 150 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4457.0:32 27.17 27:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) 2024-01-21T04:37:14.303 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:04:42.210 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:14.343 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:04:42.219 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:14.367 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:04:42.274 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:15.318 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:15.312+0000 12a57640 -1 osd.6 150 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4457.0:32 27.17 27:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: Running main() from gmock_main.cc 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] Global test environment set-up. 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1483 ms) 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (248 ms) 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (146519 ms) 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-01-21T04:37:17.045 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:08.872373+0000 osd.6 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi069-40903-1' : 1 ]) 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:10.063874+0000 osd.6 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi069-40903-1' : 1 ]) 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:10.426077+0000 mon.a [WRN] Health check update: Reduced data availability: 15 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:11.137929+0000 osd.6 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi069-40903-1' : 1 ]) 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:11.820559+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 35583.903ms) (OSD_SLOW_PING_TIME_BACK) 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:12.019998+0000 mon.b [INF] from='client.? 172.21.15.69:0/450804338' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:12.068041+0000 client.admin [INF] onexx 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:12.074820+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:12.907663+0000 osd.3 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'SimpleWritesmithi069-40829-2' : 1 ]) 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:13.216183+0000 osd.6 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi069-40903-1' : 1 ]) 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:13.886250+0000 osd.3 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'SimpleWritesmithi069-40829-2' : 1 ]) 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:14.261582+0000 osd.6 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi069-40903-1' : 1 ]) 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:14.488705+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:15.313400+0000 osd.6 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'ListObjectssmithi069-40903-1' : 1 ]) 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:15.763671+0000 mon.a [INF] from='client.? 172.21.15.69:0/3154696663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-40484-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:16.545115+0000 mon.a [WRN] Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-21T04:37:17.046 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:16.591153+0000 mon.a [INF] from='client.? 172.21.15.69:0/3154696663' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi069-40484-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-01-21T04:37:17.047 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T04:37:16.774440+0000 mon.b [INF] from='client.? 172.21.15.69:0/450804338' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-01-21T04:37:17.106 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:17.047+0000 12a57640 -1 osd.3 150 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:18.491 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:18.235+0000 12a57640 -1 osd.3 151 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:19.150 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: got: 2024-01-21T0 api_stat_pp: Running main() from gmock_main.cc 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: seed 40284 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (40414 ms) 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (14602 ms) 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (13 ms) 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-01-21T04:37:19.151 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (17500 ms) 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (16460 ms) 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (88989 ms total) 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (7627 ms) 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (20 ms) 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (14612 ms) 2024-01-21T04:37:19.152 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-01-21T04:37:19.153 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (32068 ms) 2024-01-21T04:37:19.153 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (54327 ms total) 2024-01-21T04:37:19.153 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: 2024-01-21T04:37:19.153 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-01-21T04:37:19.153 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (155544 ms total) 2024-01-21T04:37:19.153 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-01-21T04:37:19.559 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:19.482+0000 12a57640 -1 osd.3 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:19.572 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:19.384+0000 12a57640 -1 osd.6 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4478.0:12 34.10 34:0840012b:::10.obj:head [write 0~157 in=157b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-21T04:37:20.358 INFO:tasks.workunit.client.0.smithi069.stdout:4:37:16.810174+0000 client.admin [INF] threexx 2024-01-21T04:37:20.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8451 ms) 2024-01-21T04:37:20.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (156701 ms total) 2024-01-21T04:37:20.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: 2024-01-21T04:37:20.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [----------] Global test environment tear-down 2024-01-21T04:37:20.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (156701 ms total) 2024-01-21T04:37:20.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd: [ PASSED ] 4 tests. 2024-01-21T04:37:20.493 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:20.421+0000 12a57640 -1 osd.6 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4478.0:12 34.10 34:0840012b:::10.obj:head [write 0~157 in=157b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-21T04:37:20.579 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:20.562+0000 12a57640 -1 osd.3 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:21.938 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:21.928+0000 12a57640 -1 osd.6 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4478.0:12 34.10 34:0840012b:::10.obj:head [write 0~157 in=157b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-21T04:37:22.200 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:22.031+0000 12a57640 -1 osd.3 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:23.273 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T04:37:23.255+0000 12a57640 -1 osd.6 156 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4478.0:12 34.10 34:0840012b:::10.obj:head [write 0~157 in=157b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) 2024-01-21T04:37:23.336 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:23.299+0000 12a57640 -1 osd.3 155 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:24.656 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:24.597+0000 12a57640 -1 osd.3 157 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:26.134 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:26.102+0000 12a57640 -1 osd.3 158 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4681.0:2 79.0 79:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-01-21T04:37:32.032 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:31.989+0000 12a57640 -1 osd.3 164 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:10 75.11 75:897590de:test-rados-api-smithi069-40164-7::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e129) 2024-01-21T04:37:34.510 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:02.417 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:34.534 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:05:02.409 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:34.536 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:02.443 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:34.536 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:05:02.411 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:34.543 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:02.450 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:34.546 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:02.453 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:34.556 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:34.532+0000 12a57640 -1 osd.3 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:10 75.11 75:897590de:test-rados-api-smithi069-40164-7::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e129) 2024-01-21T04:37:34.746 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:02.652 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:34.748 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:02.655 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:34.757 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:05:02.632 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:37:36.842 INFO:tasks.workunit.client.0.smithi069.stdout: cls: Running main() from gmock_main.cc 2024-01-21T04:37:36.842 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-01-21T04:37:36.842 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [----------] Global test environment set-up. 2024-01-21T04:37:36.842 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-01-21T04:37:36.842 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-01-21T04:37:36.842 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [ OK ] NeoRadosCls.DNE (48324 ms) 2024-01-21T04:37:36.843 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-01-21T04:37:36.843 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (124677 ms) 2024-01-21T04:37:36.843 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [----------] 2 tests from NeoRadosCls (173001 ms total) 2024-01-21T04:37:36.843 INFO:tasks.workunit.client.0.smithi069.stdout: cls: 2024-01-21T04:37:36.843 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [----------] Global test environment tear-down 2024-01-21T04:37:36.843 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [==========] 2 tests from 1 test suite ran. (173001 ms total) 2024-01-21T04:37:36.843 INFO:tasks.workunit.client.0.smithi069.stdout: cls: [ PASSED ] 2 tests. 2024-01-21T04:37:38.512 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:38.367+0000 12a57640 -1 osd.3 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:10 75.11 75:897590de:test-rados-api-smithi069-40164-7::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e129) 2024-01-21T04:37:39.722 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T04:37:39.617+0000 12a57640 -1 osd.3 170 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:10 75.11 75:897590de:test-rados-api-smithi069-40164-7::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e129) 2024-01-21T04:38:00.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-01-21T04:38:00.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-01-21T04:38:00.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-01-21T04:38:00.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-01-21T04:38:00.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: seed 39960 2024-01-21T04:38:00.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-01-21T04:38:00.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (73615 ms) 2024-01-21T04:38:00.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-01-21T04:38:00.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (2874 ms) 2024-01-21T04:38:00.357 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (4426 ms) 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (5416 ms) 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (2033 ms) 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (13221 ms) 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (5098 ms) 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (6031 ms) 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (112714 ms total) 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-01-21T04:38:00.358 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (8120 ms) 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (2389 ms) 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (14703 ms) 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (5601 ms) 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (12014 ms) 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (5657 ms) 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (6254 ms) 2024-01-21T04:38:00.359 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-01-21T04:38:00.360 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (129 ms) 2024-01-21T04:38:00.360 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (54867 ms total) 2024-01-21T04:38:00.360 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: 2024-01-21T04:38:00.360 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-01-21T04:38:00.360 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (196875 ms total) 2024-01-21T04:38:00.360 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (679 ms) 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (457 ms) 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (197416 ms) 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (198552 ms total) 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-01-21T04:38:02.198 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (198552 ms total) 2024-01-21T04:38:02.199 INFO:tasks.workunit.client.0.smithi069.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-01-21T04:38:02.335 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: Running main() from gmock_main.cc 2024-01-21T04:38:02.335 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-01-21T04:38:02.335 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [----------] Global test environment set-up. 2024-01-21T04:38:02.335 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-01-21T04:38:02.335 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-01-21T04:38:02.335 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (6109 ms) 2024-01-21T04:38:02.335 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-01-21T04:38:02.335 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (57728 ms) 2024-01-21T04:38:02.335 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-01-21T04:38:02.335 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (134664 ms) 2024-01-21T04:38:02.336 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [----------] 3 tests from NeoRadosCmd (198501 ms total) 2024-01-21T04:38:02.336 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: 2024-01-21T04:38:02.336 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [----------] Global test environment tear-down 2024-01-21T04:38:02.336 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [==========] 3 tests from 1 test suite ran. (198501 ms total) 2024-01-21T04:38:02.336 INFO:tasks.workunit.client.0.smithi069.stdout: cmd: [ PASSED ] 3 tests. 2024-01-21T04:38:02.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-01-21T04:38:02.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-01-21T04:38:02.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-01-21T04:38:02.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-01-21T04:38:02.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-01-21T04:38:02.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (34765 ms) 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (25798 ms) 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (23249 ms) 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (12073 ms) 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (10862 ms) 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (3615 ms) 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-01-21T04:38:02.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (5473 ms) 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (54411 ms) 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5805 ms) 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (192 ms) 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (6710 ms) 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (1495 ms) 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-01-21T04:38:02.724 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (511 ms) 2024-01-21T04:38:02.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-01-21T04:38:02.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (350 ms) 2024-01-21T04:38:02.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-01-21T04:38:02.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (194 ms) 2024-01-21T04:38:02.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-01-21T04:38:02.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (139 ms) 2024-01-21T04:38:02.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (185642 ms total) 2024-01-21T04:38:02.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: 2024-01-21T04:38:02.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-01-21T04:38:02.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (198971 ms total) 2024-01-21T04:38:02.725 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-01-21T04:38:15.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-01-21T04:38:15.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] Global test environment set-up. 2024-01-21T04:38:15.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 7 tests from LibRadosList 2024-01-21T04:38:15.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-01-21T04:38:15.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjects (181345 ms) 2024-01-21T04:38:15.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-01-21T04:38:15.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (5175 ms) 2024-01-21T04:38:15.791 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo2,foo3 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo2 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo3 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo4,foo5 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo4 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo5 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo6,foo7 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo7 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo6 2024-01-21T04:38:15.792 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo4 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo5 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo7 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo6 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo1 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo1 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo2 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo3 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (7168 ms) 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 1 0 2024-01-21T04:38:15.793 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 10 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 13 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 7 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 14 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 0 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 15 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 11 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 5 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 8 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 6 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 3 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 4 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 12 0 2024-01-21T04:38:15.794 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 9 0 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2 0 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (8426 ms) 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: x cursor=MIN 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=1 cursor=7:02547ec2:::1:head 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=10 cursor=7:52ea6a34:::10:head 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=13 cursor=7:566253c9:::13:head 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=7 cursor=7:5c6b0b28:::7:head 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=14 cursor=7:62a1935d:::14:head 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=0 cursor=7:6cac518f:::0:head 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=15 cursor=7:863748b0:::15:head 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=11 cursor=7:89d3ae78:::11:head 2024-01-21T04:38:15.795 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=5 cursor=7:b29083e3:::5:head 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=8 cursor=7:bd63b0f1:::8:head 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=6 cursor=7:c4fdafeb:::6:head 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=3 cursor=7:cfc208b3:::3:head 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=4 cursor=7:d83876eb:::4:head 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=12 cursor=7:de5d7c5f:::12:head 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=9 cursor=7:e960b815:::9:head 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > oid=2 cursor=7:f905c69b:::2:head 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: FIRST> seek to MIN oid=1 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=1 cursor=7:02547ec2:::1:head 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=10 cursor=7:52ea6a34:::10:head 2024-01-21T04:38:15.796 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-01-21T04:38:15.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-01-21T04:38:15.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=13 cursor=7:566253c9:::13:head 2024-01-21T04:38:15.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:566253c9:::13:head 2024-01-21T04:38:15.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-01-21T04:38:15.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=7 cursor=7:5c6b0b28:::7:head 2024-01-21T04:38:15.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-01-21T04:38:15.797 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-01-21T04:38:18.414 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:46.321 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:18.428 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:46.335 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:18.647 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:05:46.554 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:18.810 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:46.716 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:18.820 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:46.727 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:19.024 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:05:46.931 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:19.176 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:05:47.051 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:19.192 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:05:47.067 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:19.396 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:05:47.272 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:20.187 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=14 cursor=7:62a1935d:::14:head 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=0 cursor=7:6cac518f:::0:head 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=15 cursor=7:863748b0:::15:head 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:863748b0:::15:head 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=11 cursor=7:89d3ae78:::11:head 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=5 cursor=7:b29083e3:::5:head 2024-01-21T04:38:20.188 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=8 cursor=7:bd63b0f1:::8:head 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=6 cursor=7:c4fdafeb:::6:head 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=3 cursor=7:cfc208b3:::3:head 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=4 cursor=7:d83876eb:::4:head 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-01-21T04:38:20.189 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=12 cursor=7:de5d7c5f:::12:head 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=9 cursor=7:e960b815:::9:head 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:e960b815:::9:head 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : oid=2 cursor=7:f905c69b:::2:head 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:52ea6a34:::10:head expected=7:52ea6a34:::10:head 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=10 expected=10 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-01-21T04:38:20.190 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:f905c69b:::2:head expected=7:f905c69b:::2:head 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=2 expected=2 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:e960b815:::9:head 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:e960b815:::9:head expected=7:e960b815:::9:head 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=9 expected=9 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:de5d7c5f:::12:head expected=7:de5d7c5f:::12:head 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=12 expected=12 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:d83876eb:::4:head expected=7:d83876eb:::4:head 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=4 expected=4 2024-01-21T04:38:20.191 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:5c6b0b28:::7:head expected=7:5c6b0b28:::7:head 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=7 expected=7 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:566253c9:::13:head 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:566253c9:::13:head expected=7:566253c9:::13:head 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=13 expected=13 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:cfc208b3:::3:head expected=7:cfc208b3:::3:head 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=3 expected=3 2024-01-21T04:38:20.192 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-01-21T04:38:36.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[40614]: starting. 2024-01-21T04:38:36.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[40614]: creating pool ceph_test_rados_list_parallel.smithi069-40549 2024-01-21T04:38:36.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[40614]: created object 0... 2024-01-21T04:38:36.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[40614]: created object 25... 2024-01-21T04:38:36.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[40614]: created object 49... 2024-01-21T04:38:36.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[40614]: finishing. 2024-01-21T04:38:36.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_1_[40614]: shutting down. 2024-01-21T04:38:36.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[40615]: starting. 2024-01-21T04:38:36.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[40615]: listing objects. 2024-01-21T04:38:36.894 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[40615]: listed object 0... 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[40615]: listed object 25... 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[40615]: saw 50 objects 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_2_[40615]: shutting down. 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[60085]: starting. 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[60085]: creating pool ceph_test_rados_list_parallel.smithi069-40549 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[60085]: created object 0... 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[60085]: created object 25... 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[60085]: created object 49... 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[60085]: finishing. 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_3_[60085]: shutting down. 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[60086]: starting. 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[60086]: listing objects. 2024-01-21T04:38:36.895 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[60086]: listed object 0... 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[60086]: listed object 25... 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[60086]: saw 46 objects 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_4_[60086]: shutting down. 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[60087]: starting. 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[60087]: removed 25 objects... 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[60087]: removed half of the objects 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[60087]: removed 50 objects... 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[60087]: removed 50 objects 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_5_[60087]: shutting down. 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:38:36.896 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[73746]: starting. 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[73746]: creating pool ceph_test_rados_list_parallel.smithi069-40549 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[73746]: created object 0... 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[73746]: created object 25... 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[73746]: created object 49... 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[73746]: finishing. 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_6_[73746]: shutting down. 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[73747]: starting. 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[73747]: listing objects. 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[73747]: listed object 0... 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[73747]: listed object 25... 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[73747]: listed object 50... 2024-01-21T04:38:36.897 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[73747]: saw 54 objects 2024-01-21T04:38:36.898 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_7_[73747]: shutting down. 2024-01-21T04:38:36.898 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:38:36.898 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:38:36.898 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[73748]: starting. 2024-01-21T04:38:36.898 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[73748]: added 25 objects... 2024-01-21T04:38:36.898 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[73748]: added half of the objects 2024-01-21T04:38:36.898 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[73748]: added 50 objects... 2024-01-21T04:38:36.898 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[73748]: added 50 objects 2024-01-21T04:38:36.898 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_8_[73748]: shutting down. 2024-01-21T04:38:36.898 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:38:36.898 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:38:43.884 INFO:tasks.ceph.mon.b.smithi144.stderr:2024-01-21T04:38:43.849+0000 ff82640 -1 mon.b@1(peon).paxos(paxos updating c 1..564) lease_expire from mon.0 v2:172.21.15.69:3300/0 is 0.541610837s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-01-21T04:38:49.265 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ********************** open_pools_parallel: process_1_[40700]: starting. 2024-01-21T04:38:49.265 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[40700]: creating pool ceph_test_rados_open_pools_parallel.smithi069-40643 2024-01-21T04:38:49.265 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[40700]: created object 0... 2024-01-21T04:38:49.265 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[40700]: created object 25... 2024-01-21T04:38:49.265 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[40700]: created object 49... 2024-01-21T04:38:49.265 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[40700]: finishing. 2024-01-21T04:38:49.265 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_1_[40700]: shutting down. 2024-01-21T04:38:49.265 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[40702]: starting. 2024-01-21T04:38:49.265 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[40702]: rados_pool_create. 2024-01-21T04:38:49.265 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[40702]: rados_ioctx_create. 2024-01-21T04:38:49.265 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_2_[40702]: shutting down. 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[69666]: starting. 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[69666]: creating pool ceph_test_rados_open_pools_parallel.smithi069-40643 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[69666]: created object 0... 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[69666]: created object 25... 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[69666]: created object 49... 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[69666]: finishing. 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_3_[69666]: shutting down. 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[69667]: starting. 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[69667]: rados_pool_create. 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[69667]: rados_ioctx_create. 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: process_4_[69667]: shutting down. 2024-01-21T04:38:49.266 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2024-01-21T04:38:49.267 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******************************* 2024-01-21T04:38:49.267 INFO:tasks.workunit.client.0.smithi069.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-01-21T04:38:50.209 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[40758]: starting. 2024-01-21T04:38:50.209 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[40758]: creating pool ceph_test_rados_delete_pools_parallel.smithi069-40684 2024-01-21T04:38:50.209 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[40758]: created object 0... 2024-01-21T04:38:50.209 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[40758]: created object 25... 2024-01-21T04:38:50.209 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[40758]: created object 49... 2024-01-21T04:38:50.209 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[40758]: finishing. 2024-01-21T04:38:50.209 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_1_[40758]: shutting down. 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[40761]: starting. 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[40761]: deleting pool ceph_test_rados_delete_pools_parallel.smithi069-40684 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_2_[40761]: shutting down. 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[70008]: starting. 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[70008]: creating pool ceph_test_rados_delete_pools_parallel.smithi069-40684 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[70008]: created object 0... 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[70008]: created object 25... 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[70008]: created object 49... 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[70008]: finishing. 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_3_[70008]: shutting down. 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[70010]: starting. 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[70010]: listing objects. 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[70010]: listed object 0... 2024-01-21T04:38:50.210 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[70010]: listed object 25... 2024-01-21T04:38:50.211 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[70010]: saw 50 objects 2024-01-21T04:38:50.211 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_5_[70010]: shutting down. 2024-01-21T04:38:50.211 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2024-01-21T04:38:50.211 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[70009]: starting. 2024-01-21T04:38:50.211 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[70009]: deleting pool ceph_test_rados_delete_pools_parallel.smithi069-40684 2024-01-21T04:38:50.211 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: process_4_[70009]: shutting down. 2024-01-21T04:38:50.211 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2024-01-21T04:38:50.211 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******************************* 2024-01-21T04:38:50.211 INFO:tasks.workunit.client.0.smithi069.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-01-21T04:38:51.004 INFO:tasks.workunit.client.0.smithi069.stdout:2cebd43d70 2024-01-21T04:38:51.004 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd3b190 2024-01-21T04:38:51.004 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd44b90 2024-01-21T04:38:51.004 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd3c030 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd3af90 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd3cef0 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd3bdc0 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd3dd70 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd3ccd0 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: waiting 0x562cebd3ec30 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: done. 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (37995 ms) 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (37995 ms total) 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-01-21T04:38:51.005 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.Exec (77111 ms) 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (5746 ms) 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (43262 ms) 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4365 ms) 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (73358 ms) 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] 8 tests from LibRadosMisc (203842 ms total) 2024-01-21T04:38:51.006 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: 2024-01-21T04:38:51.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [----------] Global test environment tear-down 2024-01-21T04:38:51.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (247453 ms total) 2024-01-21T04:38:51.007 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc: [ PASSED ] 12 tests. 2024-01-21T04:38:51.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: Running main() from gmock_main.cc 2024-01-21T04:38:51.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] Global test environment set-up. 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (68131 ms) 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (50901 ms) 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (3505 ms) 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.Checksum (2032 ms) 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-01-21T04:38:51.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (4424 ms) 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (3271 ms) 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (3864 ms) 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (982 ms) 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.TruncTest (2850 ms) 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (2412 ms) 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (9409 ms) 2024-01-21T04:38:51.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.RmXattr (9324 ms) 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIo.XattrIter (10136 ms) 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 14 tests from LibRadosIo (171241 ms total) 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (6582 ms) 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (9318 ms) 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (286 ms) 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (1851 ms) 2024-01-21T04:38:51.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (961 ms) 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (191 ms) 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (266 ms) 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (422 ms) 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (664 ms) 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (188 ms) 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] 10 tests from LibRadosIoEC (20729 ms total) 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: 2024-01-21T04:38:51.012 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [----------] Global test environment tear-down 2024-01-21T04:38:51.013 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [==========] 24 tests from 2 test suites ran. (247574 ms total) 2024-01-21T04:38:51.013 INFO:tasks.workunit.client.0.smithi069.stdout: api_io: [ PASSED ] 24 tests. 2024-01-21T04:38:52.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: Running main() from gmock_main.cc 2024-01-21T04:38:52.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-01-21T04:38:52.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] Global test environment set-up. 2024-01-21T04:38:52.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-01-21T04:38:52.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-01-21T04:38:52.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.Stat (46724 ms) 2024-01-21T04:38:52.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-01-21T04:38:52.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (22487 ms) 2024-01-21T04:38:52.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-01-21T04:38:52.089 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.StatNS (89398 ms) 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (50 ms) 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (20352 ms) 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 5 tests from LibRadosStat (179011 ms total) 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (13944 ms) 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (824 ms) 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-01-21T04:38:52.090 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (2472 ms) 2024-01-21T04:38:52.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-01-21T04:38:52.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1121 ms) 2024-01-21T04:38:52.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (18361 ms total) 2024-01-21T04:38:52.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: 2024-01-21T04:38:52.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [----------] Global test environment tear-down 2024-01-21T04:38:52.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (248493 ms total) 2024-01-21T04:38:52.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_stat: [ PASSED ] 9 tests. 2024-01-21T04:38:55.966 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:23.873 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:55.971 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:23.878 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:56.033 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:06:23.908 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:56.037 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:06:23.912 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:56.055 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:23.962 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:56.058 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:23.965 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:56.113 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:24.020 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:56.133 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:06:24.008 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:38:56.167 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:24.074 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:01.120 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify_test_cb 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (20589 ms) 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94621199110288 err -107 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (37482 ms) 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-01-21T04:39:01.121 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94621199891328 err -107 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (21192 ms) 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4474 notify_id 416611827712 cookie 94621199907152 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (21240 ms) 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4474 notify_id 476741369856 cookie 94621199925344 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (6078 ms) 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4474 notify_id 493921239040 cookie 94621199925344 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (6893 ms) 2024-01-21T04:39:01.122 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4474 notify_id 536870912001 cookie 94621199803600 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4474 notify_id 536870912001 cookie 94621199925280 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (8441 ms) 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4474 notify_id 592705486850 cookie 94621199925152 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4474 notify_id 592705486851 cookie 94621199925152 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (36006 ms) 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94621199925152 err -107 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_cb from 4474 notify_id 743029342210 cookie 94621199925152 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (12951 ms) 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-01-21T04:39:01.123 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94621199925152 err -107 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (12669 ms) 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (183541 ms total) 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: watch_notify_test_cb 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (11698 ms) 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (11698 ms total) 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-01-21T04:39:01.124 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (257497 ms total) 2024-01-21T04:39:01.125 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-01-21T04:39:04.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: Running main() from gmock_main.cc 2024-01-21T04:39:04.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-01-21T04:39:04.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] Global test environment set-up. 2024-01-21T04:39:04.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-01-21T04:39:04.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: seed 39922 2024-01-21T04:39:04.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-01-21T04:39:04.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-01-21T04:39:04.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-01-21T04:39:04.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (32175 ms) 2024-01-21T04:39:04.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-01-21T04:39:04.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (11640 ms) 2024-01-21T04:39:04.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-01-21T04:39:04.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (51438 ms) 2024-01-21T04:39:04.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-01-21T04:39:04.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (5795 ms) 2024-01-21T04:39:04.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-01-21T04:39:04.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2478 ms) 2024-01-21T04:39:04.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-01-21T04:39:04.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2895 ms) 2024-01-21T04:39:04.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-01-21T04:39:04.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (1536 ms) 2024-01-21T04:39:04.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-01-21T04:39:04.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5119 ms) 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (17232 ms) 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2161 ms) 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (6658 ms) 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (10701 ms) 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (15297 ms) 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (3599 ms) 2024-01-21T04:39:04.094 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (4858 ms) 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (10170 ms) 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (93 ms) 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (66 ms) 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (85 ms) 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (183996 ms total) 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-01-21T04:39:04.095 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-01-21T04:39:04.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (10665 ms) 2024-01-21T04:39:04.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-01-21T04:39:04.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (11121 ms) 2024-01-21T04:39:04.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-01-21T04:39:04.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (658 ms) 2024-01-21T04:39:04.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-01-21T04:39:04.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (74 ms) 2024-01-21T04:39:04.096 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-01-21T04:39:07.713 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIo********* 2024-01-21T04:39:07.713 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[84278]: starting. 2024-01-21T04:39:07.713 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[84278]: creating pool ceph_test_rados_list_parallel.smithi069-40549 2024-01-21T04:39:07.713 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[84278]: created object 0... 2024-01-21T04:39:07.713 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[84278]: created object 25... 2024-01-21T04:39:07.713 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[84278]: created object 49... 2024-01-21T04:39:07.714 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[84278]: finishing. 2024-01-21T04:39:07.714 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_9_[84278]: shutting down. 2024-01-21T04:39:07.714 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.714 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.714 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.714 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[84279]: starting. 2024-01-21T04:39:07.714 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[84279]: listing objects. 2024-01-21T04:39:07.714 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[84279]: listed object 0... 2024-01-21T04:39:07.714 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[84279]: listed object 25... 2024-01-21T04:39:07.715 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[84279]: listed object 50... 2024-01-21T04:39:07.715 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[84279]: listed object 75... 2024-01-21T04:39:07.715 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[84279]: saw 98 objects 2024-01-21T04:39:07.715 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_10_[84279]: shutting down. 2024-01-21T04:39:07.715 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.715 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.715 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.715 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[84280]: starting. 2024-01-21T04:39:07.715 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[84280]: added 25 objects... 2024-01-21T04:39:07.716 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[84280]: added half of the objects 2024-01-21T04:39:07.716 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[84280]: added 50 objects... 2024-01-21T04:39:07.716 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[84280]: added 50 objects 2024-01-21T04:39:07.716 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_11_[84280]: shutting down. 2024-01-21T04:39:07.716 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.716 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.716 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.716 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[84282]: starting. 2024-01-21T04:39:07.716 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[84282]: removed 25 objects... 2024-01-21T04:39:07.717 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[84282]: removed half of the objects 2024-01-21T04:39:07.717 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[84282]: removed 50 objects... 2024-01-21T04:39:07.717 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[84282]: removed 50 objects 2024-01-21T04:39:07.717 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_13_[84282]: shutting down. 2024-01-21T04:39:07.717 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.717 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.718 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.718 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[84281]: starting. 2024-01-21T04:39:07.718 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[84281]: added 25 objects... 2024-01-21T04:39:07.718 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[84281]: added half of the objects 2024-01-21T04:39:07.718 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[84281]: added 50 objects... 2024-01-21T04:39:07.718 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[84281]: added 50 objects 2024-01-21T04:39:07.718 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_12_[84281]: shutting down. 2024-01-21T04:39:07.719 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.719 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.719 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.719 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.719 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[98161]: starting. 2024-01-21T04:39:07.719 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[98161]: creating pool ceph_test_rados_list_parallel.smithi069-40549 2024-01-21T04:39:07.719 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[98161]: created object 0... 2024-01-21T04:39:07.720 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[98161]: created object 25... 2024-01-21T04:39:07.720 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[98161]: created object 49... 2024-01-21T04:39:07.720 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[98161]: finishing. 2024-01-21T04:39:07.720 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_14_[98161]: shutting down. 2024-01-21T04:39:07.720 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.720 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.720 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.720 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:07.721 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[98162]: starting. 2024-01-21T04:39:07.721 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[98162]: listing objects. 2024-01-21T04:39:07.721 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[98162]: listed object 0... 2024-01-21T04:39:16.456 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:44.362 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:16.470 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:44.377 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:16.539 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:06:44.413 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:16.542 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:06:44.417 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:16.552 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:44.459 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:16.554 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:44.461 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:16.604 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:06:44.511 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:16.618 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[98162]: listed object 2 api_lock: Running main() from gmock_main.cc 2024-01-21T04:39:16.618 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-01-21T04:39:16.618 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] Global test environment set-up. 2024-01-21T04:39:16.618 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (30665 ms) 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockShared (30393 ms) 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (6074 ms) 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (7960 ms) 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (13102 ms) 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4720 ms) 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-01-21T04:39:16.619 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (85707 ms) 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (1071 ms) 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLock (179692 ms total) 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (8656 ms) 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (2865 ms) 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (3010 ms) 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-01-21T04:39:16.620 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (2993 ms) 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (739 ms) 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (2263 ms) 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (6659 ms) 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (598 ms) 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (27783 ms total) 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [----------] Global test environment tear-down 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (273157 ms total) 2024-01-21T04:39:16.621 INFO:tasks.workunit.client.0.smithi069.stdout: api_lock: [ PASSED ] 16 tests. 2024-01-21T04:39:16.623 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:06:44.498 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:16.665 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:06:44.571 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:40.014 INFO:tasks.workunit.client.0.smithi069.stdout:5... 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[98162]: listed object 50... 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[98162]: listed object 75... 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[98162]: listed object 100... 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[98162]: listed object 125... 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[98162]: saw 150 objects 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_15_[98162]: shutting down. 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[98163]: starting. 2024-01-21T04:39:40.015 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[98163]: added 25 objects... 2024-01-21T04:39:40.016 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[98163]: added half of the objects 2024-01-21T04:39:40.016 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[98163]: added 50 objects... 2024-01-21T04:39:40.016 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[98163]: added 50 objects 2024-01-21T04:39:40.016 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: process_16_[98163]: shutting down. 2024-01-21T04:39:40.016 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:40.016 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:40.016 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:40.016 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:40.016 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******************************* 2024-01-21T04:39:40.016 INFO:tasks.workunit.client.0.smithi069.stdout: list_parallel: ******* SUCCESS ********** 2024-01-21T04:39:41.551 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:09.458 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:41.555 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:09.462 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:41.623 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:09.530 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:41.630 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:09.537 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:41.634 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:09.509 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:41.642 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:09.517 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:41.668 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:09.575 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:41.733 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:09.608 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:41.749 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:09.656 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:39:45.629 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: Running main() from gmock_main.cc 2024-01-21T04:39:45.630 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-01-21T04:39:45.630 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [----------] Global test environment set-up. 2024-01-21T04:39:45.630 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-01-21T04:39:45.630 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-01-21T04:39:45.630 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (28143 ms) 2024-01-21T04:39:45.630 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-01-21T04:39:45.630 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (55845 ms) 2024-01-21T04:39:45.630 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-01-21T04:39:45.630 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (82893 ms) 2024-01-21T04:39:45.630 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-01-21T04:39:45.630 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (14285 ms) 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (33378 ms) 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (30689 ms) 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (7383 ms) 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (4440 ms) 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (4735 ms) 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (10590 ms) 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-01-21T04:39:45.631 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (10110 ms) 2024-01-21T04:39:45.632 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-01-21T04:39:45.632 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (9533 ms) 2024-01-21T04:39:45.632 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-01-21T04:39:45.632 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (5023 ms) 2024-01-21T04:39:45.632 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-01-21T04:39:45.632 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (4640 ms) 2024-01-21T04:39:45.632 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (301687 ms total) 2024-01-21T04:39:45.632 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: 2024-01-21T04:39:45.632 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [----------] Global test environment tear-down 2024-01-21T04:39:45.632 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (301687 ms total) 2024-01-21T04:39:45.632 INFO:tasks.workunit.client.0.smithi069.stdout: read_operations: [ PASSED ] 14 tests. 2024-01-21T04:39:45.634 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-01-21T04:39:45.634 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-01-21T04:39:45.634 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-01-21T04:39:45.634 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: seed 40192 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (45276 ms) 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (16383 ms) 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (10111 ms) 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (36089 ms) 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (22407 ms) 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (130266 ms total) 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-01-21T04:39:45.635 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (8676 ms) 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (57437 ms) 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (43385 ms) 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (9990 ms) 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (3492 ms) 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-766-g0edf41a6/rpm/el9/BUILD/ceph-19.0.0-766-g0edf41a6/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-01-21T04:39:45.636 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi069-40192-7 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: waiting for snaps to purge 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (22317 ms) 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (145297 ms total) 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (6 ms) 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5429 ms) 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5435 ms total) 2024-01-21T04:39:45.637 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-01-21T04:39:45.638 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-01-21T04:39:45.638 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-01-21T04:39:45.638 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (5448 ms) 2024-01-21T04:39:45.638 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-01-21T04:39:47.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LiECPP.RoundTripPP2 (3225 ms) 2024-01-21T04:39:47.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-01-21T04:39:47.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (1320 ms) 2024-01-21T04:39:47.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-01-21T04:39:47.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (6391 ms) 2024-01-21T04:39:47.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-01-21T04:39:47.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (4150 ms) 2024-01-21T04:39:47.721 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (2287 ms) 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (2240 ms) 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3762 ms) 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (1258 ms) 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (10336 ms) 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4550 ms) 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (1883 ms) 2024-01-21T04:39:47.722 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-01-21T04:39:47.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (1781 ms) 2024-01-21T04:39:47.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-01-21T04:39:47.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (909 ms) 2024-01-21T04:39:47.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (66610 ms total) 2024-01-21T04:39:47.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: 2024-01-21T04:39:47.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [----------] Global test environment tear-down 2024-01-21T04:39:47.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (304274 ms total) 2024-01-21T04:39:47.723 INFO:tasks.workunit.client.0.smithi069.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-01-21T04:39:51.195 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: Running main() from gmock_main.cc 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] Global test environment set-up. 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (76118 ms) 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (23792 ms) 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (11259 ms) 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (11939 ms) 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (27659 ms) 2024-01-21T04:39:51.196 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (150767 ms total) 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (10409 ms) 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (40761 ms) 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (58649 ms) 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (16816 ms) 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (7351 ms) 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-01-21T04:39:51.197 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi069-41077-11. 2024-01-21T04:39:51.198 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: Waiting for snaps to purge. 2024-01-21T04:39:51.198 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (22482 ms) 2024-01-21T04:39:51.198 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (156469 ms total) 2024-01-21T04:39:51.198 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: 2024-01-21T04:39:51.198 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [----------] Global test environment tear-down 2024-01-21T04:39:51.198 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (307236 ms total) 2024-01-21T04:39:51.198 INFO:tasks.workunit.client.0.smithi069.stdout: snapshots: [ PASSED ] 11 tests. 2024-01-21T04:39:55.352 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: Running main() from gmock_main.cc 2024-01-21T04:39:55.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-01-21T04:39:55.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] Global test environment set-up. 2024-01-21T04:39:55.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-01-21T04:39:55.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-01-21T04:39:55.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (11307 ms) 2024-01-21T04:39:55.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-01-21T04:39:55.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (11052 ms) 2024-01-21T04:39:55.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-01-21T04:39:55.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (31733 ms) 2024-01-21T04:39:55.353 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (7409 ms) 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (61501 ms total) 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (103228 ms) 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (35333 ms) 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (39495 ms) 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (178056 ms total) 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-01-21T04:39:55.354 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (5681 ms) 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (4868 ms) 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (7991 ms) 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2838 ms) 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (21378 ms total) 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (18018 ms) 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-01-21T04:39:55.355 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (6367 ms) 2024-01-21T04:39:55.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (24385 ms total) 2024-01-21T04:39:55.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: 2024-01-21T04:39:55.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [----------] Global test environment tear-down 2024-01-21T04:39:55.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (311783 ms total) 2024-01-21T04:39:55.356 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-01-21T04:40:01.802 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:29.709 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:01.806 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:29.713 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:01.851 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:29.726 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:01.852 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:29.727 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:01.873 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:29.780 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:01.876 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:29.783 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:01.887 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:29.791 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:01.923 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:29.798 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:01.947 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:29.854 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: Running main() from gmock_main.cc 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: [----------] Global test environment set-up. 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: [----------] 14 tests from NeoRadosIo 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.Limits (78409 ms) 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (116253 ms) 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.ReadOp (30885 ms) 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-01-21T04:40:02.316 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.SparseRead (28224 ms) 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.RoundTrip (8076 ms) 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (7540 ms) 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (7335 ms) 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (10121 ms) 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (5272 ms) 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.Trunc (4991 ms) 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-01-21T04:40:02.317 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.Remove (6696 ms) 2024-01-21T04:40:02.318 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-01-21T04:40:02.318 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (5468 ms) 2024-01-21T04:40:02.318 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-01-21T04:40:02.318 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.RmXattr (4523 ms) 2024-01-21T04:40:02.318 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-01-21T04:40:02.318 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ OK ] NeoRadosIo.GetXattrs (4643 ms) 2024-01-21T04:40:02.318 INFO:tasks.workunit.client.0.smithi069.stdout: io: [----------] 14 tests from NeoRadosIo (318436 ms total) 2024-01-21T04:40:02.318 INFO:tasks.workunit.client.0.smithi069.stdout: io: 2024-01-21T04:40:02.318 INFO:tasks.workunit.client.0.smithi069.stdout: io: [----------] Global test environment tear-down 2024-01-21T04:40:02.318 INFO:tasks.workunit.client.0.smithi069.stdout: io: [==========] 14 tests from 1 test suite ran. (318436 ms total) 2024-01-21T04:40:02.318 INFO:tasks.workunit.client.0.smithi069.stdout: io: [ PASSED ] 14 tests. 2024-01-21T04:40:03.672 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:31.579 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:03.674 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:31.581 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:03.698 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:31.605 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:03.710 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:31.585 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:03.711 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:31.586 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:03.737 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:31.612 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:04.224 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:32.130 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:04.224 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:32.131 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:04.246 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:32.153 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:19.461 INFO:tasks.workunit.client.0.smithi069.stdout:bRadosSnapshotsECPP.SnapRemovePP (3206 ms) 2024-01-21T04:40:19.461 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-01-21T04:40:19.461 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (5489 ms) 2024-01-21T04:40:19.461 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-01-21T04:40:19.461 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (3171 ms) 2024-01-21T04:40:19.461 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (17314 ms total) 2024-01-21T04:40:19.461 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (6611 ms) 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4390 ms) 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4092 ms) 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (15093 ms total) 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (335885 ms total) 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-01-21T04:40:19.462 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-01-21T04:40:19.463 INFO:tasks.workunit.client.0.smithi069.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-01-21T04:40:22.106 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:50.013 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:22.112 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:50.019 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:22.172 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:50.047 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:22.175 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:50.050 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:22.184 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:07:50.091 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:22.194 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:50.101 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:22.198 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:50.104 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:22.266 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:07:50.141 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:22.267 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:07:50.174 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:28.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: Running main() from gmock_main.cc 2024-01-21T04:40:28.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-01-21T04:40:28.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] Global test environment set-up. 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4893 ms) 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (73956 ms) 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (46052 ms) 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (10686 ms) 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (9072 ms) 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (22753 ms) 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-01-21T04:40:28.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (17706 ms) 2024-01-21T04:40:28.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-01-21T04:40:28.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (30001 ms) 2024-01-21T04:40:28.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-01-21T04:40:28.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (32454 ms) 2024-01-21T04:40:28.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-01-21T04:40:28.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (14721 ms) 2024-01-21T04:40:28.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-01-21T04:40:28.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (8574 ms) 2024-01-21T04:40:28.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-01-21T04:40:28.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (4226 ms) 2024-01-21T04:40:28.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-01-21T04:40:28.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (7922 ms) 2024-01-21T04:40:28.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-01-21T04:40:28.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (4274 ms) 2024-01-21T04:40:28.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-01-21T04:40:28.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.Flush (10269 ms) 2024-01-21T04:40:28.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-01-21T04:40:28.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (6706 ms) 2024-01-21T04:40:28.091 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (5453 ms) 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (4598 ms) 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (4571 ms) 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3425 ms) 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3532 ms) 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3674 ms) 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3969 ms) 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-01-21T04:40:28.092 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3469 ms) 2024-01-21T04:40:28.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-01-21T04:40:28.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3887 ms) 2024-01-21T04:40:28.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-01-21T04:40:28.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3818 ms) 2024-01-21T04:40:28.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 26 tests from LibRadosAio (344661 ms total) 2024-01-21T04:40:28.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: 2024-01-21T04:40:28.093 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-01-21T04:40:33.518 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.Sim list: Running main() from gmock_main.cc 2024-01-21T04:40:33.518 INFO:tasks.workunit.client.0.smithi069.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-01-21T04:40:33.518 INFO:tasks.workunit.client.0.smithi069.stdout: list: [----------] Global test environment set-up. 2024-01-21T04:40:33.518 INFO:tasks.workunit.client.0.smithi069.stdout: list: [----------] 3 tests from NeoradosList 2024-01-21T04:40:33.518 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-01-21T04:40:33.518 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ OK ] NeoradosList.ListObjects (155239 ms) 2024-01-21T04:40:33.518 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-01-21T04:40:33.519 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ OK ] NeoradosList.ListObjectsNS (46269 ms) 2024-01-21T04:40:33.519 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-01-21T04:40:33.519 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ OK ] NeoradosList.ListObjectsMany (148120 ms) 2024-01-21T04:40:33.519 INFO:tasks.workunit.client.0.smithi069.stdout: list: [----------] 3 tests from NeoradosList (349628 ms total) 2024-01-21T04:40:33.519 INFO:tasks.workunit.client.0.smithi069.stdout: list: 2024-01-21T04:40:33.519 INFO:tasks.workunit.client.0.smithi069.stdout: list: [----------] Global test environment tear-down 2024-01-21T04:40:33.519 INFO:tasks.workunit.client.0.smithi069.stdout: list: [==========] 3 tests from 1 test suite ran. (349628 ms total) 2024-01-21T04:40:33.519 INFO:tasks.workunit.client.0.smithi069.stdout: list: [ PASSED ] 3 tests. 2024-01-21T04:40:37.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:c4fdafeb:::6:head expected=7:c4fdafeb:::6:head 2024-01-21T04:40:37.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-01-21T04:40:37.181 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=6 expected=6 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:bd63b0f1:::8:head expected=7:bd63b0f1:::8:head 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=8 expected=8 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:02547ec2:::1:head expected=7:02547ec2:::1:head 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=1 expected=1 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:b29083e3:::5:head expected=7:b29083e3:::5:head 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=5 expected=5 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-01-21T04:40:37.182 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:89d3ae78:::11:head expected=7:89d3ae78:::11:head 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=11 expected=11 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:863748b0:::15:head 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:863748b0:::15:head expected=7:863748b0:::15:head 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=15 expected=15 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:6cac518f:::0:head expected=7:6cac518f:::0:head 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=0 expected=0 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : cursor()=7:62a1935d:::14:head expected=7:62a1935d:::14:head 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-01-21T04:40:37.183 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: : entry=14 expected=14 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (11977 ms) 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (103623 ms) 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 0/5 -> MIN 7:33333333::::head 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 1/5 -> 7:33333333::::head 7:66666666::::head 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 2/5 -> 7:66666666::::head 7:99999999::::head 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 3/5 -> 7:99999999::::head 7:cccccccc::::head 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: split 4/5 -> 7:cccccccc::::head MAX 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (18445 ms) 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 7 tests from LibRadosList (336159 ms total) 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-01-21T04:40:37.184 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (2031 ms) 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo2,foo3 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo2 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo3 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo1,foo4,foo5 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo4 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo5 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo1 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset foo6,foo7 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo7 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: foo6 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-01-21T04:40:37.185 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo4 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo5 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo7 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns2:foo6 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: ns1:foo1 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo1 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo2 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: :foo3 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (3166 ms) 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 1 0 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 10 0 2024-01-21T04:40:37.186 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 13 0 2024-01-21T04:40:42.125 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:10.032 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:42.127 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:10.034 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:42.183 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:10.090 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:42.186 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:10.093 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:42.188 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:10.095 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:42.191 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:08:10.065 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:42.193 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:08:10.068 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:42.243 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:10.150 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:42.250 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:08:10.125 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:40:44.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 7 0 2024-01-21T04:40:44.919 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 14 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 0 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 15 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 11 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 5 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 8 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 6 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 3 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 4 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 12 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 9 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2 0 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-01-21T04:40:44.920 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (4493 ms) 2024-01-21T04:40:44.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 3 tests from LibRadosListEC (9690 ms total) 2024-01-21T04:40:44.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2024-01-21T04:40:44.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-01-21T04:40:44.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-01-21T04:40:44.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3594 ms) 2024-01-21T04:40:44.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] 1 test from LibRadosListNP (3594 ms total) 2024-01-21T04:40:44.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: 2024-01-21T04:40:44.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [----------] Global test environment tear-down 2024-01-21T04:40:44.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [==========] 11 tests from 3 test suites ran. (361450 ms total) 2024-01-21T04:40:44.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_list: [ PASSED ] 11 tests. 2024-01-21T04:40:51.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-01-21T04:40:51.866 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (22680 ms) 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (52627 ms) 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-01-21T04:40:51.867 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (51990 ms) 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (23719 ms) 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (13725 ms) 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (20047 ms) 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (183366 ms) 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (368154 ms total) 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (368154 ms total) 2024-01-21T04:40:51.868 INFO:tasks.workunit.client.0.smithi069.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-01-21T04:41:02.428 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:30.334 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:02.472 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:30.378 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:02.527 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:08:30.402 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:02.531 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:30.438 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:02.540 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:08:30.415 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:02.552 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:30.459 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:02.586 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:30.493 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:02.612 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:08:30.487 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:02.636 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:30.543 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:22.385 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:50.291 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:22.392 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:50.299 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:22.437 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:50.344 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:22.440 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:50.346 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:22.441 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:08:50.316 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:22.444 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:08:50.319 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:22.466 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:08:50.372 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:22.493 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:08:50.368 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:22.500 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:08:50.407 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:30.693 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: Running main() from gmock_main.cc 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [----------] Global test environment set-up. 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (123140 ms) 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (94679 ms) 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (188964 ms) 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [----------] 3 tests from NeoradosECList (406783 ms total) 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [----------] Global test environment tear-down 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (406783 ms total) 2024-01-21T04:41:30.694 INFO:tasks.workunit.client.0.smithi069.stdout: ec_list: [ PASSED ] 3 tests. 2024-01-21T04:41:42.518 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:10.424 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:42.523 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:10.429 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:42.579 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:10.454 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:42.579 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:10.485 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:42.582 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:10.488 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:42.585 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:10.460 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:42.592 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:10.499 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:42.652 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:10.527 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:42.657 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:10.564 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:52.395 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:20.301 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:52.397 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:20.304 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:52.444 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:20.351 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:52.467 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:20.342 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:52.507 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:20.345 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:52.507 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:20.379 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:52.507 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:20.383 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:52.521 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:20.396 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:41:52.523 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:20.430 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:02.887 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:30.793 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:02.890 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:30.797 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:02.927 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:30.834 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:02.936 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:30.811 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:02.941 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:30.816 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:02.945 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:30.852 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:02.955 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:30.861 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:02.979 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:30.853 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:02.989 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:30.895 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:06.371 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-01-21T04:42:06.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-01-21T04:42:06.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-01-21T04:42:06.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-01-21T04:42:06.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-01-21T04:42:06.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (5050 ms) 2024-01-21T04:42:06.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-01-21T04:42:06.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (347187 ms) 2024-01-21T04:42:06.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-01-21T04:42:06.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (7938 ms) 2024-01-21T04:42:06.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-01-21T04:42:06.372 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3263 ms) 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2612 ms) 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3537 ms) 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3375 ms) 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3439 ms) 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3722 ms) 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3400 ms) 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (4296 ms) 2024-01-21T04:42:06.373 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3255 ms) 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (2916 ms) 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (4405 ms) 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3152 ms) 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3499 ms) 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3426 ms) 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3125 ms) 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3520 ms) 2024-01-21T04:42:06.374 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3527 ms) 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3123 ms) 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3405 ms) 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3295 ms) 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3309 ms) 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3622 ms) 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3095 ms) 2024-01-21T04:42:06.375 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-01-21T04:42:22.833 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:50.740 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:22.836 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:50.743 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:22.886 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:50.793 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:22.888 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:50.763 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:22.889 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:50.796 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:22.892 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:09:50.799 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:22.893 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:50.767 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:22.951 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:09:50.826 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:22.969 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:09:50.876 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:32.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAi api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-01-21T04:42:32.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-01-21T04:42:32.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-01-21T04:42:32.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-01-21T04:42:32.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-01-21T04:42:32.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-01-21T04:42:32.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-01-21T04:42:32.301 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: seed 40122 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (35 ms) 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (109299 ms) 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (10859 ms) 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (13540 ms) 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (2217 ms) 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (10761 ms) 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-01-21T04:42:32.302 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (1036 ms) 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (3045 ms) 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (630 ms) 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (18791 ms) 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (544 ms) 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (2385 ms) 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (1628 ms) 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: osd_max_attr_size = 0 2024-01-21T04:42:32.303 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (196580 ms) 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (9454 ms) 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: done waiting, doing copies 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: waiting for final deep scrubs... 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: done waiting 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (76732 ms) 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (74 ms) 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (41 ms) 2024-01-21T04:42:32.304 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-01-21T04:42:32.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (3820 ms) 2024-01-21T04:42:32.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-01-21T04:42:32.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-01-21T04:42:32.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-01-21T04:42:32.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-01-21T04:42:32.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-01-21T04:42:32.305 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-01-21T04:42:35.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [pleWrite 2024-01-21T04:42:35.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7585 ms) 2024-01-21T04:42:35.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-01-21T04:42:35.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7932 ms) 2024-01-21T04:42:35.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-01-21T04:42:35.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (8309 ms) 2024-01-21T04:42:35.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-01-21T04:42:35.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7926 ms) 2024-01-21T04:42:35.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7099 ms) 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (9636 ms) 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (8421 ms) 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7937 ms) 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7719 ms) 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7071 ms) 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (8542 ms) 2024-01-21T04:42:35.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6887 ms) 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7881 ms) 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6879 ms) 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (8628 ms) 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (8940 ms) 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (127392 ms total) 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [----------] Global test environment tear-down 2024-01-21T04:42:35.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (472053 ms total) 2024-01-21T04:42:35.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio: [ PASSED ] 42 tests. 2024-01-21T04:42:42.954 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:10:10.828 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:42.957 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:10:10.832 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:42.962 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:10.869 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:42.964 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:10.871 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:43.029 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:10:10.904 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:43.035 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:10.942 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:43.039 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:10.946 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:43.041 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:10.948 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:42:43.182 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:11.088 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:03.104 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:31.010 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:03.117 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:31.024 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:03.119 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:31.026 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:03.155 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:31.062 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:03.176 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:10:31.051 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:03.188 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:10:31.063 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:03.251 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:31.157 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:03.252 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:10:31.127 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:03.262 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:31.168 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:06.579 INFO:tasks.workunit.client.0.smithi069.stdout:----------] 22 tests from LibRadosMiscPP (461471 ms total) 2024-01-21T04:43:06.579 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-01-21T04:43:06.579 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-01-21T04:43:06.579 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-01-21T04:43:06.579 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (6967 ms) 2024-01-21T04:43:06.579 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (6967 ms total) 2024-01-21T04:43:06.579 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (885 ms) 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (58 ms) 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (943 ms total) 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (1512 ms) 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-01-21T04:43:06.580 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (43 ms) 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (1555 ms total) 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (597 ms) 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (184 ms) 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (781 ms total) 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1923 ms) 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1923 ms total) 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: 2024-01-21T04:43:06.581 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-01-21T04:43:06.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (503018 ms total) 2024-01-21T04:43:06.582 INFO:tasks.workunit.client.0.smithi069.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-01-21T04:43:23.145 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:51.052 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:23.149 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:51.056 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:23.195 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:10:51.070 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:23.197 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:51.103 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:23.197 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:10:51.072 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:23.199 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:51.105 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:23.210 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:10:51.117 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:23.241 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:10:51.115 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:23.247 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:10:51.154 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:40.744 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:08.651 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:40.747 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:08.654 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:40.799 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:08.674 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:40.801 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:08.708 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:40.803 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:08.710 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:40.804 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:08.679 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:40.810 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:08.717 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:40.848 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:08.723 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:40.875 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:08.782 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:43.378 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:11.284 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:43.448 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:11.354 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:43.488 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:11.394 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:43.511 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:11.418 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:43.511 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:11.386 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:43.516 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:11.423 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:43.529 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:11.404 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:43.544 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:11.451 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:43:43.549 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:11.424 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:03.432 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:31.338 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:03.498 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:31.405 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:03.558 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:31.465 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:03.562 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:31.437 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:03.566 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:31.473 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:03.575 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:31.450 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:03.611 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:31.518 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:03.632 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:31.539 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:03.641 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:31.516 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:23.559 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:51.465 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:23.563 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:51.438 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:23.566 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:51.441 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:23.568 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:51.475 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:23.612 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:11:51.487 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:23.642 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:11:51.548 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:23.644 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:51.551 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:23.645 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:51.552 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:23.690 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:11:51.597 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:38.625 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:06.531 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:38.630 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:06.537 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:38.672 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:06.578 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:38.677 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:06.552 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:38.678 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:06.553 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:38.685 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:06.591 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:38.687 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:06.594 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:38.720 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:06.595 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:38.724 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:06.631 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:43.710 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:11.616 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:43.760 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:11.635 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:43.766 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:11.641 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:43.772 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:11.679 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:43.790 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:11.665 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:43.811 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:11.718 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:43.841 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:11.748 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:43.845 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:11.752 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:43.861 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:11.768 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:49.202 INFO:tasks.workunit.client.0.smithi069.stdout:o.RoundTripAppendPP (4446 ms) 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3583 ms) 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3110 ms) 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3394 ms) 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: setting pool EIO 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: max_success 36, min_failed 37 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (6932 ms) 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4384 ms) 2024-01-21T04:44:49.203 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (464343 ms total) 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3475 ms) 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9130 ms) 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (18296 ms) 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3488 ms) 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (34389 ms total) 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3233 ms) 2024-01-21T04:44:49.204 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3233 ms total) 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13804 ms) 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (8087 ms) 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6964 ms) 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7785 ms) 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3523 ms) 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7290 ms) 2024-01-21T04:44:49.205 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6739 ms) 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7180 ms) 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6680 ms) 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (8398 ms) 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7062 ms) 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (8055 ms) 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (8973 ms) 2024-01-21T04:44:49.206 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-01-21T04:44:53.832 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:21.738 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:53.881 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:21.756 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:53.883 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:21.758 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:53.913 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:21.820 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:53.915 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:21.790 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:53.950 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:21.857 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:53.954 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:21.860 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:53.957 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:21.864 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:44:53.993 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:21.900 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:04.092 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:31.999 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:04.093 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:31.968 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:04.095 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:32.002 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:04.098 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:31.973 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:04.135 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:32.042 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:04.146 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:32.053 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:04.157 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:32.031 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:04.174 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:32.081 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:04.238 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:32.145 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:08.946 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:36.852 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:08.948 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:36.855 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:08.969 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:36.876 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:08.994 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:36.869 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:08.995 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:36.870 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:09.008 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:36.915 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:09.009 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:36.916 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:09.012 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:36.887 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:09.029 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:36.936 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:24.139 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:52.045 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:24.144 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:52.051 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:24.194 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:12:52.101 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:24.205 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:52.080 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:24.209 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:52.085 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:24.213 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:52.119 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:24.219 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:52.126 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:24.262 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:12:52.137 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:24.270 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:12:52.177 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:35.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3266 ms) 2024-01-21T04:45:35.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-01-21T04:45:35.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7941 ms) 2024-01-21T04:45:35.309 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7026 ms) 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7934 ms) 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7268 ms) 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7879 ms) 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (8056 ms) 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (149910 ms total) 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (651875 ms total) 2024-01-21T04:45:35.310 INFO:tasks.workunit.client.0.smithi069.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-01-21T04:45:38.972 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:06.879 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:38.974 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:06.881 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:39.015 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:06.921 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:39.018 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:06.893 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:39.021 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:06.896 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:39.022 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:06.929 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:39.025 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:06.932 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:39.059 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:06.934 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:39.064 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:06.971 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:44.114 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:12.021 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:44.116 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:12.023 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:44.132 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:12.039 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:44.168 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:12.075 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:44.173 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:12.048 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:44.173 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:12.080 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:44.176 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:12.051 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:44.191 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:12.098 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:44.198 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:12.073 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:53.979 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:21.885 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:54.026 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:21.933 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:54.054 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:21.961 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:54.072 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:21.947 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:54.076 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:21.951 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:54.082 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:21.988 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:54.083 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:21.990 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:54.100 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:21.975 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:45:54.112 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:22.019 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-01-21T04:46:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-01-21T04:46:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-01-21T04:46:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: seed 40131 2024-01-21T04:46:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-01-21T04:46:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (71101 ms) 2024-01-21T04:46:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-01-21T04:46:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (26998 ms) 2024-01-21T04:46:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-01-21T04:46:02.520 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (5086 ms) 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (103185 ms total) 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (74968 ms) 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (74522 ms) 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (52037 ms) 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [3] 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [4,3] 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [5,4,3] 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-01-21T04:46:02.521 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting some heads 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 6 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 5 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 4 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: promoting from clones for snap 3 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrubs... 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (181004 ms) 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (12689 ms) 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (9848 ms) 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (10153 ms) 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-01-21T04:46:02.522 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (9627 ms) 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (11858 ms) 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9955 ms) 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (11864 ms) 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (46756 ms) 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (9510 ms) 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (11729 ms) 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (17167 ms) 2024-01-21T04:46:02.523 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-01-21T04:46:02.524 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (10254 ms) 2024-01-21T04:46:02.524 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-01-21T04:46:02.524 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (12431 ms) 2024-01-21T04:46:04.051 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:31.957 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:04.053 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:31.960 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:04.078 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:31.985 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:04.100 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:31.975 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:04.101 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:31.976 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:04.104 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:32.011 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:04.105 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:32.012 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:04.125 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:32.000 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:04.133 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:32.040 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:14.355 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:42.230 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:14.358 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:42.233 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:14.363 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:42.270 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:14.387 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:42.294 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:14.432 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:42.339 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:14.433 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:42.307 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:14.434 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:42.341 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:14.447 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:42.354 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:14.471 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:42.378 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:24.203 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:52.110 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:24.204 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:52.111 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:24.226 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:13:52.133 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:24.254 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:52.129 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:24.255 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:52.131 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:24.269 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:52.176 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:24.271 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:52.178 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:24.279 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:13:52.154 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:24.294 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:13:52.201 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:24.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-01-21T04:46:24.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: hmm, no HitSet yet 2024-01-21T04:46:24.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: ok, hit_set contains 266:602f83fe:::foo:head 2024-01-21T04:46:24.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (10017 ms) 2024-01-21T04:46:24.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-01-21T04:46:24.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg_num = 32 2024-01-21T04:46:24.474 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 0 ls 1705812379,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 1 ls 1705812379,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 2 ls 1705812379,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 3 ls 1705812380,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 4 ls 1705812379,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 5 ls 1705812379,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 6 ls 1705812380,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 7 ls 1705812379,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 8 ls 1705812380,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 9 ls 1705812380,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 10 ls 1705812380,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 11 ls 1705812379,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 12 ls 1705812380,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 13 ls 1705812380,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 14 ls 1705812379,0 2024-01-21T04:46:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 15 ls 1705812379,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 16 ls 1705812380,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 17 ls 1705812379,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 18 ls 1705812379,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 19 ls 1705812379,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 20 ls 1705812379,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 21 ls 1705812380,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 22 ls 1705812379,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 23 ls 1705812379,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 24 ls 1705812380,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 25 ls 1705812379,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 26 ls 1705812379,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 27 ls 1705812379,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 28 ls 1705812380,0 2024-01-21T04:46:24.476 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 29 ls 1705812379,0 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 30 ls 1705812379,0 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg 31 ls 1705812379,0 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: pg_num = 32 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:6cac518f:::0:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:02547ec2:::1:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:f905c69b:::2:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:cfc208b3:::3:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:d83876eb:::4:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:b29083e3:::5:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:c4fdafeb:::6:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:5c6b0b28:::7:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:bd63b0f1:::8:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:e960b815:::9:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:52ea6a34:::10:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:89d3ae78:::11:head 2024-01-21T04:46:24.477 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:de5d7c5f:::12:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:566253c9:::13:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:62a1935d:::14:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:863748b0:::15:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:3958e169:::16:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:4d4dabf9:::17:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:8391935d:::18:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:28883081:::19:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:69259c59:::20:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:4bdb80b7:::21:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:a11c5d71:::22:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:271af37b:::23:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:95b121be:::24:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:58d1031b:::25:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:0a050783:::26:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:c709704c:::27:head 2024-01-21T04:46:24.478 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:cbe56eaf:::28:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:86b4b162:::29:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:70d89383:::30:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:dd450c7c:::31:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:6d5729b1:::32:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:c388f3fb:::33:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:56cfea31:::34:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:9dbc1bf7:::35:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:40b74ccd:::36:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:4d5aaf42:::37:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:920f362c:::38:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:6cc53222:::39:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:9cad833f:::40:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:1ea84d41:::41:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:c4480ef6:::42:head 2024-01-21T04:46:24.479 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:a694361e:::43:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:d1bd33e9:::44:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:ddc2cd5d:::45:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:2b782207:::46:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:7b187fca:::47:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:90ecdf6f:::48:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:a5ed95fe:::49:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:ea0eaa55:::50:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:f33ef17b:::51:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:a0d1b2f6:::52:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:60c5229e:::53:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:edcbc575:::54:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:102cf253:::55:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:efb7fb0b:::56:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:50d0a326:::57:head 2024-01-21T04:46:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:d4dc5daf:::58:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:3a130462:::59:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:ec87ed71:::60:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:d5bc9454:::61:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:3ddfe313:::62:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:7c2816b9:::63:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:47e00e4d:::64:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:c6410c18:::65:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:b48ed237:::66:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:cd63ad31:::67:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:b179e92b:::68:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:0d9f741a:::69:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:6d3352ae:::70:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:c6d5c19e:::71:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:bc4729c3:::72:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:77e930b9:::73:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:0abeecfd:::74:head 2024-01-21T04:46:24.481 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:b7c37e15:::75:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:b6378398:::76:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:02bd68de:::77:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:cc795d2d:::78:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:630d4fea:::79:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:e0d29ef5:::80:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:fd6f13d2:::81:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:606461d5:::82:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:eadbdc43:::83:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:8761d0bb:::84:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:9ef0186f:::85:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:e0d41294:::86:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:961de695:::87:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:1423148f:::88:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:633a8fa2:::89:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:a8653809:::90:head 2024-01-21T04:46:24.482 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:3dac8b33:::91:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:35aad435:::92:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:f6dcc343:::93:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:dbbdad87:::94:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:1cb48ce0:::95:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:03cd461c:::96:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:17a4ea99:::97:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:9993c9a7:::98:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:6394211c:::99:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:94c7ae57:::100:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:6fdee5bb:::101:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:9a477fd1:::102:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:eb850916:::103:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:affc56b9:::104:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:b42dc814:::105:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:f319f8f0:::106:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:9a40b9de:::107:head 2024-01-21T04:46:24.483 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:8b524f28:::108:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:e3de589f:::109:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:90f90a5b:::110:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:a7b4f1d7:::111:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:af51766e:::112:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:b6f90bd1:::113:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:e0261208:::114:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:c9569ef7:::115:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:61bebe50:::116:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:fe93412b:::117:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:d3d38bee:::118:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:3100ba0c:::119:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:d0560ada:::120:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:f0ea8b35:::121:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:766f231a:::122:head 2024-01-21T04:46:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:a07a2582:::123:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:bd7c6b3a:::124:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:fb2ddaff:::125:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:4408e1fe:::126:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:ee1df7a7:::127:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:c3002909:::128:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:4f48ffa9:::129:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:edf38733:::130:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:c08425c0:::131:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:5f902d98:::132:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:41ea2c93:::133:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:813cee13:::134:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:0131818d:::135:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:26ba5a85:::136:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:381b8a5a:::137:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:28797e47:::138:head 2024-01-21T04:46:24.485 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:bfca7f22:::139:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:36807075:::140:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:80b03975:::141:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:5c15709b:::142:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:f39ea15e:::143:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:ea992956:::144:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:48887b1c:::145:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:9f24a9dd:::146:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:987f100b:::147:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:d2dd3581:::148:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:7fed1808:::149:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:c80b70e9:::150:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:85ed90f9:::151:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:36428b24:::152:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:d044c34a:::153:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:7c18bf58:::154:head 2024-01-21T04:46:24.486 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:d1c21232:::155:head 2024-01-21T04:46:24.487 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:a7a3c575:::156:head 2024-01-21T04:46:24.487 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:87da0633:::157:head 2024-01-21T04:46:24.487 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:d5ac3822:::158:head 2024-01-21T04:46:24.487 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:3f20522d:::159:head 2024-01-21T04:46:24.487 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:6ca26563:::160:head 2024-01-21T04:46:24.487 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:532ce135:::161:head 2024-01-21T04:46:24.487 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:c78863e6:::162:head 2024-01-21T04:46:44.286 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:12.193 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:44.288 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:12.195 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:44.323 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:12.229 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:44.337 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:12.244 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:44.338 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:12.213 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:44.339 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:12.214 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:44.340 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:12.247 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:44.375 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:12.250 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:44.389 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:12.296 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:45.173 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:13.048 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:45.174 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:13.049 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:45.182 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:13.057 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:45.235 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:13.142 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:45.236 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:13.143 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:45.238 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:13.145 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:45.239 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:13.146 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:45.245 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:13.152 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:45.248 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:13.155 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:59.327 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:27.234 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:59.331 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:27.237 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:59.353 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:27.260 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:59.374 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:27.281 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:59.376 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:27.251 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:59.377 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:27.284 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:59.380 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:27.255 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:59.403 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:27.278 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:46:59.403 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:27.310 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:04.470 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:32.377 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:04.473 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:32.380 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:04.500 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:32.407 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:04.519 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:32.394 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:04.522 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:32.397 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:04.527 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:32.434 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:04.532 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:32.439 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:04.538 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:32.414 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:04.557 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:32.464 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:14.385 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:42.292 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:14.388 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:42.295 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:14.420 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:42.327 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:14.434 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:42.309 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:14.435 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:42.342 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:14.437 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:42.312 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:14.439 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:42.346 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:14.463 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:42.338 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:14.466 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:42.373 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:24.419 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:52.325 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:24.420 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:52.327 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:24.441 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:52.348 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:24.462 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:52.337 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:24.463 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:52.370 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:24.465 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:52.341 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:24.468 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:52.375 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:24.486 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:52.361 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:24.490 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:52.397 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:29.429 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:57.336 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:29.434 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:57.341 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:29.448 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:14:57.355 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:29.479 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:57.354 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:29.481 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:57.357 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:29.485 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:57.392 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:29.486 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:57.393 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:29.493 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:14:57.368 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:29.498 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:14:57.405 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:33.921 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:7462ddf6:::163:head 2024-01-21T04:47:33.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:5d165639:::164:head 2024-01-21T04:47:33.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:f43765fc:::165:head 2024-01-21T04:47:33.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:b4c720e9:::166:head 2024-01-21T04:47:33.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:e694b040:::167:head 2024-01-21T04:47:33.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:afa38db2:::168:head 2024-01-21T04:47:33.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:77ba9f53:::169:head 2024-01-21T04:47:33.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:87495034:::170:head 2024-01-21T04:47:33.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:7c96bf0e:::171:head 2024-01-21T04:47:33.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:dbe346cc:::172:head 2024-01-21T04:47:33.922 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:e943ec24:::173:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:f97a9c0c:::174:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:6f26e74d:::175:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:4f95e106:::176:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:0e6f2f8f:::177:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:05db05f1:::178:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:38a78d66:::179:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:d095610b:::180:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:a1a9d709:::181:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:1e5d39db:::182:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:f7df4fb9:::183:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:03a7f161:::184:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:ba70721e:::185:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:28e5662d:::186:head 2024-01-21T04:47:33.923 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:973d52de:::187:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:4303eb1c:::188:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:b990b48e:::189:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:29b8165b:::190:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:3547f197:::191:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:7e260936:::192:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:1abec7b1:::193:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:10fdda93:::194:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:15817eea:::195:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:770bab57:::196:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:ed9e13e7:::197:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:71471a8f:::198:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: checking for 267:10fb1d02:::199:head 2024-01-21T04:47:33.924 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (20803 ms) 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first is 1705812401 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,1705812404,1705812405,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,1705812404,1705812405,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,1705812404,1705812405,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,1705812404,1705812405,1705812407,1705812408,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,1705812404,1705812405,1705812407,1705812408,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,1705812404,1705812405,1705812407,1705812408,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,1705812404,1705812405,1705812407,1705812408,1705812410,1705812411,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,1705812404,1705812405,1705812407,1705812408,1705812410,1705812411,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812401,1705812404,1705812405,1705812407,1705812408,1705812410,1705812411,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705812405,1705812407,1705812408,1705812410,1705812411,1705812413,1705812414,0 2024-01-21T04:47:33.925 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first now 1705812405, trimmed 2024-01-21T04:47:33.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22803 ms) 2024-01-21T04:47:33.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-01-21T04:47:33.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: foo0 2024-01-21T04:47:33.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-01-21T04:47:33.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (15281 ms) 2024-01-21T04:47:33.926 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-01-21T04:47:38.988 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:06.895 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:39.048 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:06.955 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:39.052 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:06.927 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:39.054 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:06.930 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:39.083 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:06.990 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:39.088 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:06.963 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:39.111 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:07.018 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:39.114 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:07.021 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:39.155 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:07.062 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:44.511 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:12.418 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:44.514 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:12.421 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:44.540 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:12.447 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:44.568 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:12.475 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:44.569 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:12.444 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:44.571 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:12.446 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:44.573 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:12.480 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:44.587 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:12.462 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:44.591 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:12.498 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:49.636 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:17.543 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:49.637 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:17.544 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:49.652 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:17.559 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:49.683 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:17.558 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:49.684 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:17.559 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:49.695 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:17.602 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:49.696 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:17.604 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:49.697 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:17.572 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:49.711 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:17.618 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:59.584 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:27.491 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:59.591 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:27.498 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:59.624 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:27.531 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:59.650 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:27.557 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:59.655 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:27.562 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:59.661 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:27.536 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:59.666 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:27.541 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:59.687 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:27.594 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:47:59.694 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:27.569 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:09.748 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:37.654 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:09.812 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:37.719 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:09.853 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:37.760 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:09.871 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:37.746 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:09.877 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:37.752 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:09.892 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:37.799 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:09.897 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:37.804 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:09.918 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:37.793 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:09.938 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:37.845 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:14.837 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:42.744 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:14.846 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:42.753 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:14.874 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:42.781 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:14.889 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:42.796 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:14.909 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:42.784 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:14.913 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:42.788 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:14.927 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:42.834 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:14.936 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:42.812 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:14.950 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:42.857 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:29.716 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:57.622 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:29.720 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:57.627 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:29.772 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:57.647 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:29.774 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:57.649 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:29.775 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:57.682 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:29.785 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:57.693 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:29.791 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:57.698 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:29.817 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:57.692 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:29.840 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:57.747 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:30.427 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:58.302 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:30.428 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:58.304 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:30.436 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:15:58.311 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:30.459 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:58.366 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:30.460 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:58.367 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:30.469 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:15:58.376 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:30.492 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:58.399 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:30.493 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:58.401 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:30.504 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:15:58.412 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:44.760 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:12.666 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:44.763 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:12.670 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:44.786 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:12.693 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:44.808 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:12.683 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:44.811 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:12.686 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:44.813 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:12.720 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:44.817 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:12.724 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:44.833 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:12.708 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:44.849 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:12.756 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:49.939 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:17.814 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:49.939 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:17.846 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:49.942 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:17.817 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:49.944 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:17.851 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:49.957 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:17.832 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:49.960 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:17.867 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:49.998 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:17.905 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:49.999 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:17.906 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:48:50.014 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:17.921 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:05.048 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:32.955 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:05.051 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:32.926 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:05.057 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:32.932 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:05.058 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:32.965 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:05.067 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:32.974 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:05.070 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:32.977 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:05.085 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:32.960 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:05.099 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:33.006 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:05.103 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:33.010 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:09.891 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:37.798 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:09.893 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:37.800 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:09.910 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:37.817 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:09.939 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:37.846 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:09.944 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:37.851 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:09.947 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:37.822 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:09.950 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:37.825 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:09.964 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:37.871 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:09.973 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:37.848 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:19.972 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:47.879 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:19.981 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:47.856 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:19.982 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:47.857 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:19.985 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:47.892 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:19.991 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:47.898 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:20.050 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:47.957 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:20.056 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:47.931 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:20.072 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:47.978 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:20.073 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:47.980 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:30.200 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:58.106 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:30.205 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:58.112 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:30.206 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:58.082 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:30.213 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:58.088 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:30.250 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:16:58.157 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:30.251 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:16:58.126 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:30.269 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:58.176 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:30.281 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:58.188 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:30.328 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:16:58.235 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:35.170 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:03.076 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:35.171 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:03.046 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:35.173 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:03.080 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:35.198 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:03.105 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:35.240 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:03.146 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:35.241 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:03.149 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:35.260 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:03.135 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:35.263 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:03.170 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:35.290 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:03.165 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:50.103 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:18.010 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:50.107 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:18.014 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:50.125 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:18.032 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:50.150 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:18.026 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:50.153 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:18.028 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:50.155 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:18.062 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:50.156 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:18.063 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:50.177 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:18.052 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:50.178 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:18.085 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:55.049 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:22.956 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:55.055 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:22.961 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:55.066 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:22.973 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:55.100 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:23.007 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:55.101 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:22.976 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:55.104 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:22.979 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:55.114 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:22.989 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:55.118 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:23.025 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:55.129 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:23.036 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:58.132 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:26.039 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:58.135 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:26.042 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:58.148 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:26.056 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:58.183 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:26.059 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:58.184 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:26.060 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:58.188 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:26.095 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:58.189 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:26.096 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:58.198 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:26.073 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:49:58.204 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:26.111 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:10.105 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:38.012 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:10.107 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:38.014 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:10.142 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:38.049 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:10.162 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:38.037 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:10.164 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:38.039 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:10.166 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:38.073 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:10.169 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:38.077 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:10.188 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:38.063 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:10.197 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:38.104 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:15.070 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:42.976 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:15.115 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:43.023 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:15.130 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:43.037 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:15.165 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:43.040 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:15.165 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:43.073 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:15.168 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:43.043 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:15.168 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:43.075 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:15.180 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:43.056 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:15.183 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:43.090 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:25.237 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:53.144 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:25.292 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:53.199 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:25.293 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:53.200 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:25.294 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:53.202 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:25.305 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:53.180 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:25.315 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:53.190 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:25.320 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:17:53.227 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:25.349 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:17:53.224 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:25.378 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:17:53.285 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:35.390 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:03.297 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:35.393 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:03.300 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:35.432 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:03.339 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:35.442 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:03.317 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:35.443 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:03.318 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:35.457 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:03.364 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:35.458 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:03.365 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:35.475 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:03.350 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:35.483 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:03.391 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:40.249 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:08.156 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:40.251 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:08.158 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:40.264 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:08.171 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:40.294 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:08.170 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:40.298 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:08.173 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:40.300 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:08.207 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:40.302 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:08.209 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:40.309 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:08.184 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:40.314 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:08.221 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:54.210 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:50:54.196+0000 22278640 -1 snap_mapper.add_oid found existing snaps mapped on 285:e1a7adc4:test-rados-api-smithi069-40131-80::foo:2, removing 2024-01-21T04:50:54.224 INFO:tasks.ceph.osd.7.smithi144.stderr:2024-01-21T04:50:54.217+0000 2c28c640 -1 snap_mapper.add_oid found existing snaps mapped on 285:e1a7adc4:test-rados-api-smithi069-40131-80::foo:2, removing 2024-01-21T04:50:55.407 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:23.314 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:55.462 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:23.337 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:55.474 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:23.339 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:55.475 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:23.383 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:55.494 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:23.369 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:55.532 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:23.439 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:55.542 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:23.449 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:55.550 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:23.457 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:50:55.576 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:23.483 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:00.318 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:28.224 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:00.322 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:28.229 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:00.339 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:28.246 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:00.367 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:28.243 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:00.370 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:28.245 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:00.371 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:28.278 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:00.372 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:28.280 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:00.384 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:28.260 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:00.388 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:28.296 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:15.368 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:43.274 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:15.372 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:43.280 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:15.407 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:43.314 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:15.416 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:43.291 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:15.419 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:43.295 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:15.420 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:43.327 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:15.424 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:43.331 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:15.453 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:43.328 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:15.464 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:43.371 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:16.331 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:44.205 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:16.331 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:44.206 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:16.340 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:44.215 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:16.438 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:44.345 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:16.439 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:44.347 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:16.449 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:44.357 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:16.475 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:44.382 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:16.476 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:44.383 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:16.485 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:44.392 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:29.040 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (22499 ms) 2024-01-21T04:51:29.040 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-01-21T04:51:29.040 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (24247 ms) 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3633 ms) 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5580 ms) 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (5372 ms) 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4505 ms) 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5562 ms) 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-01-21T04:51:29.041 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (41674 ms) 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17686 ms) 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5489 ms) 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (5570 ms) 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27074 ms) 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.042 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (17128 ms) 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (7001 ms) 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5369 ms) 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrubs... 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25306 ms) 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (11459 ms) 2024-01-21T04:51:29.043 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-01-21T04:51:29.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11374 ms) 2024-01-21T04:51:29.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-01-21T04:51:29.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:51:29.044 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11089 ms) 2024-01-21T04:51:30.431 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:58.338 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:30.437 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:58.344 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:30.474 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:18:58.381 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:30.488 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:58.363 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:30.492 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:58.367 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:30.503 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:58.410 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:30.506 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:58.413 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:30.526 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:18:58.401 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:30.537 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:18:58.444 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:35.449 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:03.355 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:35.450 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:03.358 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:35.471 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:03.378 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:35.506 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:03.381 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:35.507 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:03.414 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:35.508 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:03.383 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:35.509 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:03.416 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:35.523 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:03.399 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:35.534 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:03.441 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:45.423 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:13.329 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:45.470 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:13.377 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:45.497 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:13.404 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:45.517 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:13.424 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:45.521 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:13.428 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:45.528 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:13.403 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:45.531 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:13.407 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:45.548 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:13.455 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:45.560 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:13.435 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:55.662 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:23.569 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:55.666 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:23.574 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:55.699 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:23.606 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:55.734 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:23.641 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:55.735 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:23.611 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:55.739 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:23.646 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:55.742 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:23.617 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:55.766 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:23.674 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:51:55.771 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:23.647 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:00.636 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:28.543 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:00.643 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:28.550 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:00.657 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:28.565 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:00.688 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:28.563 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:00.689 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:28.564 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:00.694 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:28.602 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:00.697 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:28.604 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:00.704 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:28.579 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:00.715 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:28.622 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:10.054 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:37.961 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:10.067 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:37.974 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:10.068 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:37.943 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:10.106 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:38.013 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:10.108 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:37.983 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:10.112 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:38.019 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:10.132 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:38.039 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:10.133 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:38.008 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:10.138 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:38.046 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:15.700 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:43.607 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:15.702 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:43.609 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:15.718 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:43.626 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:15.751 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:43.626 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:15.752 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:43.628 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:15.756 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:43.663 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:15.762 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:43.669 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:15.768 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:43.643 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:15.779 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:43.686 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:20.728 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:48.635 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:20.730 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:48.637 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:20.742 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:19:48.649 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:20.777 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:48.652 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:20.778 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:48.653 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:20.785 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:48.692 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:20.787 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:48.694 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:20.792 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:19:48.667 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:20.803 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:19:48.710 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:35.637 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:03.544 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:35.685 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:03.592 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:35.711 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:03.618 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:35.732 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:03.640 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:35.735 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:03.611 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:35.736 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:03.643 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:35.739 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:03.614 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:35.760 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:03.668 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:35.762 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:03.638 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:36.490 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:04.365 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:36.491 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:04.366 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:36.499 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:04.374 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:36.533 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:04.440 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:36.533 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:04.441 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:36.539 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:04.446 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:36.540 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:04.447 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:36.542 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:04.449 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:36.548 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:04.455 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:36.728 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T04:52:36.721+0000 2ba8b640 -1 snap_mapper.add_oid found existing snaps mapped on 85:64323911:test-rados-api-smithi069-40131-97::foo:21, removing 2024-01-21T04:52:36.773 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T04:52:36.706+0000 22278640 -1 snap_mapper.add_oid found existing snaps mapped on 85:64323911:test-rados-api-smithi069-40131-97::foo:21, removing 2024-01-21T04:52:43.909 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFl ec_io: Running main() from gmock_main.cc 2024-01-21T04:52:43.909 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-01-21T04:52:43.909 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [----------] Global test environment set-up. 2024-01-21T04:52:43.909 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-01-21T04:52:43.909 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-01-21T04:52:43.909 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (11301 ms) 2024-01-21T04:52:43.909 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-01-21T04:52:43.909 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (1068720 ms) 2024-01-21T04:52:43.909 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (1080021 ms total) 2024-01-21T04:52:43.909 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: 2024-01-21T04:52:43.910 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [----------] Global test environment tear-down 2024-01-21T04:52:43.910 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (1080021 ms total) 2024-01-21T04:52:43.910 INFO:tasks.workunit.client.0.smithi069.stdout: ec_io: [ PASSED ] 2 tests. 2024-01-21T04:52:50.917 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:18.824 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:50.983 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:18.890 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:51.030 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:18.937 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:51.046 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:18.921 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:51.047 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:18.923 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:51.059 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:18.966 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:51.061 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:18.968 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:51.095 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:18.971 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:51.103 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:19.010 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:56.087 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:23.994 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:56.092 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:23.999 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:56.106 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:24.013 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:56.137 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:24.044 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:56.141 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:24.017 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:56.143 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:24.050 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:56.143 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:24.019 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:56.158 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:24.033 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:52:56.158 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:24.065 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:06.088 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:33.994 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:06.089 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:33.996 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:06.124 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:34.031 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:06.142 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:34.048 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:06.146 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:34.053 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:06.170 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:34.045 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:06.172 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:34.080 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:06.173 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:34.049 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:06.202 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:34.077 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:15.950 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:43.857 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:15.999 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:43.906 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:16.024 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:43.931 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:16.047 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:43.922 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:16.050 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:43.925 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:16.051 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:43.958 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:16.054 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:43.962 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:16.074 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:43.949 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:16.079 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:43.986 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:21.237 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:49.112 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:21.242 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:49.149 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:21.294 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:49.201 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:21.298 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:49.173 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:21.308 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:49.215 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:21.316 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:49.224 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:21.331 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:20:49.206 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:21.335 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:20:49.242 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:21.356 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:20:49.263 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:36.296 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:04.203 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:36.301 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:04.209 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:36.359 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:04.266 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:36.369 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:04.244 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:36.371 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:04.246 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:36.376 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:04.284 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:36.378 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:04.285 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:36.407 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:04.282 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:36.433 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:04.340 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:51.411 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:19.318 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:51.413 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:19.320 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:51.446 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:19.353 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:51.457 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:19.333 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:51.459 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:19.334 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:51.463 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:19.370 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:51.468 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:19.375 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:51.503 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:19.379 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:51.511 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:19.418 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:56.297 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:24.204 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:56.301 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:24.208 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:56.318 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:24.225 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:56.347 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:24.222 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:56.348 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:24.224 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:56.353 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:24.260 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:56.356 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:24.263 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:56.363 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:24.238 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:53:56.372 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:24.279 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:06.502 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:34.408 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:06.505 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:34.412 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:06.532 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:34.439 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:06.571 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:34.446 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:06.572 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:34.447 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:06.583 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:34.490 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:06.586 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:34.493 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:06.620 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:34.527 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:06.622 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:34.497 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:11.377 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:39.284 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:11.379 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:39.286 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:11.385 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:39.260 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:11.386 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:39.262 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:11.395 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:39.302 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:11.398 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:39.305 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:11.447 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:39.322 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:11.463 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:39.370 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:11.492 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:39.399 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:16.360 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:44.267 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:16.362 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:44.269 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:16.378 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:44.285 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:16.407 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:44.283 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:16.409 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:44.284 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:16.410 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:44.317 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:16.412 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:44.320 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:16.422 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:44.298 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:16.428 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:44.335 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:21.382 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:49.289 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:21.383 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:49.291 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:21.397 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:21:49.304 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:21.428 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:49.303 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:21.429 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:49.305 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:21.432 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:49.340 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:21.434 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:49.341 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:21.442 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:21:49.317 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:21.449 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:21:49.356 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:36.372 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:04.279 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:36.422 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:04.329 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:36.442 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:04.350 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:36.464 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:04.340 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:36.467 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:04.342 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:36.472 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:04.379 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:36.476 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:04.383 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:36.487 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:04.362 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:36.498 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:04.405 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:41.544 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:09.451 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:41.547 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:09.454 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:41.566 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:09.473 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:41.592 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:09.467 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:41.595 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:09.471 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:41.596 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:09.503 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:41.599 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:09.507 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:41.611 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:09.486 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:41.624 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:09.531 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:56.714 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:24.620 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:56.720 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:24.627 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:56.755 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:24.662 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:56.762 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:24.637 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:56.763 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:24.638 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:56.781 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:24.688 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:56.782 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:24.689 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:56.797 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:24.672 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:54:56.816 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:24.723 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:11.631 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:39.538 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:11.683 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:39.558 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:11.684 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:39.559 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:11.691 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:39.598 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:11.716 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:39.591 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:11.733 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:39.640 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:11.750 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:39.658 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:11.753 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:39.660 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:11.794 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:39.701 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:21.825 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:49.732 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:21.829 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:49.736 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:21.858 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:49.765 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:21.881 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:49.788 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:21.886 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:49.793 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:21.889 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:49.764 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:21.894 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:49.769 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:21.912 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:49.819 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:21.929 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:49.805 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:26.827 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:54.734 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:26.840 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:54.748 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:26.843 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:54.718 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:26.845 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:54.752 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:26.845 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:54.721 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:26.848 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:54.756 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:26.907 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:22:54.782 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:26.918 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:22:54.825 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:26.924 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:22:54.831 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:32.903 INFO:tasks.workunit.client.0.smithi069.stdout:ush 2024-01-21T04:55:32.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:55:32.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (11376 ms) 2024-01-21T04:55:32.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-01-21T04:55:32.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:55:32.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8591 ms) 2024-01-21T04:55:32.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-01-21T04:55:32.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:55:32.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5773 ms) 2024-01-21T04:55:32.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-01-21T04:55:32.904 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (27995 ms) 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15383 ms) 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (14761 ms) 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000010 69 73 |is| 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 00000012 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12819 ms) 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: require_osd_release = squid 2024-01-21T04:55:32.905 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6279 ms) 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (995871 ms total) 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1080 ms) 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11732 ms) 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19101 ms) 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (18 ms) 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (31931 ms total) 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-01-21T04:55:32.906 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7990 ms) 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8655 ms) 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: waiting for scrub... 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: done waiting 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (27138 ms) 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (12006 ms) 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (11099 ms) 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (9934 ms) 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11998 ms) 2024-01-21T04:55:32.907 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-01-21T04:55:41.797 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:09.704 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:41.798 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:09.705 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:41.826 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:09.733 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:41.846 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:09.722 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:41.848 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:09.723 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:41.852 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:09.759 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:41.856 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:09.763 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:41.874 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:09.749 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:41.884 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:09.791 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:42.779 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:10.654 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:42.779 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:10.655 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:42.788 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:10.663 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:42.850 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:10.757 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:42.851 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:10.758 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:42.851 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:10.759 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:42.852 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:10.759 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:42.863 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:10.770 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:42.863 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:10.770 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:56.839 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:24.746 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:56.844 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:24.751 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:56.874 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:24.781 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:56.888 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:24.795 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:56.893 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:24.768 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:56.894 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:24.801 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:56.895 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:24.770 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:56.925 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:24.832 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:55:56.927 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:24.803 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:01.855 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:29.762 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:01.870 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:29.778 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:01.889 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:29.797 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:01.906 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:29.813 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:01.906 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:29.782 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:01.908 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:29.815 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:01.908 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:29.784 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:01.922 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:29.798 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:01.929 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:29.837 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:11.330 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:39.237 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:11.389 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:39.296 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:11.424 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:39.331 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:11.453 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:39.360 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:11.454 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:39.361 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:11.466 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:39.341 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:11.469 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:39.345 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:11.484 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:39.391 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:11.503 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:39.378 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:12.518 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:40.393 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:12.519 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:40.394 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:12.528 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:40.403 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:12.568 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:40.475 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:12.569 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:40.476 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:12.578 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:40.485 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:12.649 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:40.557 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:12.651 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:40.558 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:12.661 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:40.568 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:22.084 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:49.990 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:22.139 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:50.046 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:22.140 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:50.047 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:22.144 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:50.051 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:22.150 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:50.026 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:22.155 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:50.030 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:22.175 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:50.082 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:22.187 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:50.094 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:22.197 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:50.072 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:26.929 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:54.836 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:26.981 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:54.888 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:26.998 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:23:54.905 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:27.027 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:54.934 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:27.029 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:54.905 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:27.031 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:54.938 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:27.031 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:54.906 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:27.046 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:23:54.921 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:27.054 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:23:54.961 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:42.153 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:10.060 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:42.206 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:10.113 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:42.212 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:10.087 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:42.217 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:10.093 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:42.252 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:10.159 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:42.256 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:10.131 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:42.260 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:10.167 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:42.263 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:10.170 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:42.297 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:10.204 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:43.234 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:11.109 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:43.245 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:11.121 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:43.260 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:11.136 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:43.377 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:11.284 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:43.378 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:11.285 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:43.405 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:11.312 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:43.410 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:11.317 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:43.411 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:11.319 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:43.425 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:11.332 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:57.132 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:25.039 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:57.181 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:25.088 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:57.181 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:25.088 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:57.182 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:25.089 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:57.210 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:25.117 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:57.211 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:25.119 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:57.227 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:25.102 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:57.230 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:25.105 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:56:57.254 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:25.130 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:02.189 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:30.096 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:02.192 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:30.100 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:02.220 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:30.128 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:02.237 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:30.113 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:02.241 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:30.117 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:02.243 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:30.151 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:02.245 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:30.152 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:02.253 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:30.128 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:02.256 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:30.163 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:12.300 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:40.207 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:12.302 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:40.209 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:12.333 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:40.209 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:12.335 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:40.210 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:12.337 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:40.244 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:12.338 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:40.245 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:12.341 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:40.248 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:12.362 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:40.237 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:12.381 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:40.288 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_ api_watch_notify_pp: Running main() from gmock_main.cc 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1333462 ms) 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (576 ms) 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1334038 ms total) 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: 2024-01-21T04:57:18.008 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (511 ms) 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4363 ms) 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (93 ms) 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (109 ms) 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94280124780480 notify_id 4402341478404 notifier_gid 21445 2024-01-21T04:57:18.009 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (133 ms) 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94280124780480 notify_id 4402341478405 notifier_gid 21445 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (150 ms) 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94280124780480 notify_id 4402341478406 notifier_gid 21445 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (335 ms) 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94280124780480 notify_id 4406636445703 notifier_gid 21445 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (177 ms) 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94280124780480 notify_id 4406636445704 notifier_gid 21445 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (116 ms) 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94280124799968 notify_id 4406636445705 notifier_gid 21445 2024-01-21T04:57:18.010 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (132 ms) 2024-01-21T04:57:18.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-01-21T04:57:18.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: trying... 2024-01-21T04:57:18.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94280124799968 notify_id 4406636445706 notifier_gid 21445 2024-01-21T04:57:18.011 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: timed out 2024-01-21T04:57:22.459 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:50.366 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:22.513 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:50.420 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:22.557 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:50.464 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:22.561 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:50.437 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:22.564 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:50.472 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:22.566 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:50.473 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:22.567 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:50.442 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:22.593 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:50.500 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:22.597 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:50.472 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:27.372 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:55.279 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:27.374 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:55.282 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:27.390 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:24:55.297 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:27.418 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:55.293 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:27.420 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:55.295 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:27.425 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:55.332 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:27.426 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:55.333 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:27.435 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:24:55.311 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:27.442 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:24:55.350 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:33.441 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2024-01-21T04:57:33.441 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushed 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3092 ms) 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: trying... 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94280124780480 notify_id 4415226380299 notifier_gid 21445 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: timed out 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushed 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3092 ms) 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: List watches 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94280124780480 notify_id 4423816314892 notifier_gid 21445 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 done 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: watch_check 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: unwatch2 2024-01-21T04:57:33.442 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: done 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3170 ms) 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: List watches 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: handle_notify cookie 94280124780480 notify_id 4436701216781 notifier_gid 21445 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: notify2 done 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: watch_check 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: unwatch2 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: flushing 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: done 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3336 ms) 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (18809 ms total) 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-01-21T04:57:33.443 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (1369765 ms total) 2024-01-21T04:57:33.444 INFO:tasks.workunit.client.0.smithi069.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-01-21T04:57:42.433 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:10.340 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:42.451 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:10.358 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:42.477 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:10.384 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:42.488 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:10.364 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:42.489 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:10.396 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:42.490 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:10.398 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:42.491 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:10.367 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:42.515 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:10.390 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:42.516 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:10.423 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:43.551 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:11.426 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:43.552 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:11.427 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:43.561 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:11.437 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:43.630 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:11.537 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:43.630 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:11.537 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:43.631 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:11.538 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:43.631 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:11.538 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:43.641 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:11.548 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:43.643 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:11.550 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:57.585 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:25.492 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:57.587 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:25.494 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:57.636 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:25.511 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:57.636 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:25.543 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:57.636 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:25.512 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:57.651 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:25.558 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:57.654 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:25.561 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:57.667 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:25.543 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:57:57.687 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:25.594 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:02.503 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:30.410 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:02.505 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:30.413 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:02.519 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:30.426 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:02.552 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:30.459 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:02.556 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:30.463 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:02.556 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:30.431 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:02.559 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:30.434 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:02.570 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:30.445 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:02.570 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:30.477 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:17.230 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:45.137 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:17.236 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:45.143 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:17.293 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:45.200 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:17.308 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:45.183 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:17.311 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:45.187 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:17.313 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:45.220 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:17.316 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:45.223 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:17.364 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:45.239 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:17.373 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:45.280 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:18.624 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:46.499 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:18.624 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:46.500 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:18.629 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:46.536 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:18.630 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:46.538 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:18.633 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:46.509 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:18.640 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:46.547 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:18.818 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:46.725 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:18.819 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:46.726 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:18.829 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:46.736 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:22.560 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:50.467 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:22.622 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:50.529 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:22.640 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:25:50.547 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:22.671 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:50.546 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:22.672 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:50.579 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:22.675 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:50.582 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:22.678 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:50.554 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:22.691 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:25:50.598 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:22.696 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:25:50.571 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:32.869 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:00.776 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:32.878 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:00.785 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:32.918 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:00.825 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:32.944 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:00.819 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:32.947 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:00.822 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:32.955 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:00.862 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:32.956 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:00.864 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:32.985 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:00.860 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:32.998 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:00.905 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:33.560 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:01.435 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:33.561 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:01.436 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:33.570 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:01.445 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:33.572 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:01.479 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:33.573 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:01.480 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:33.581 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:01.489 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:33.615 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:01.522 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:33.623 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:01.530 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:33.679 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:01.586 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:34.084 INFO:tasks.workunit.client.0.smithi069.stdout:tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (9373 ms) 2024-01-21T04:58:34.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-01-21T04:58:34.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (15040 ms) 2024-01-21T04:58:34.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-01-21T04:58:34.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (10080 ms) 2024-01-21T04:58:34.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-01-21T04:58:34.084 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (14417 ms) 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8496 ms) 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (10110 ms) 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: hmm, no HitSet yet 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: ok, hit_set contains 317:602f83fe:::foo:head 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9799 ms) 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,0 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first is 1705813009 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,0 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,0 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,1705813012,1705813013,0 2024-01-21T04:58:34.085 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,1705813012,1705813013,0 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,1705813012,1705813013,0 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,1705813012,1705813013,1705813015,1705813016,0 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,1705813012,1705813013,1705813015,1705813016,0 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,1705813012,1705813013,1705813015,1705813016,0 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,1705813012,1705813013,1705813015,1705813016,1705813018,1705813019,0 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,1705813012,1705813013,1705813015,1705813016,1705813018,1705813019,0 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813009,1705813012,1705813013,1705813015,1705813016,1705813018,1705813019,0 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: got ls 1705813013,1705813015,1705813016,1705813018,1705813019,1705813021,1705813022,0 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: first now 1705813013, trimmed 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (23064 ms) 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: foo0 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (18544 ms) 2024-01-21T04:58:34.086 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (22973 ms) 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (23724 ms) 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5780 ms) 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (5196 ms) 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (5407 ms) 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4518 ms) 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (275342 ms total) 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: 2024-01-21T04:58:34.087 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-01-21T04:58:34.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1430509 ms total) 2024-01-21T04:58:34.088 INFO:tasks.workunit.client.0.smithi069.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-01-21T04:58:34.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39826 2024-01-21T04:58:34.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39826 2024-01-21T04:58:34.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40020 2024-01-21T04:58:34.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40020 2024-01-21T04:58:34.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.090 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40571 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40571 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40251 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40251 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40114 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40114 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39902 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39902 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40207 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40207 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40711 2024-01-21T04:58:34.091 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40711 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40754 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40754 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39946 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39946 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40608 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40608 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40445 2024-01-21T04:58:34.092 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40445 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39797 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39797 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39862 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39862 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40472 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40472 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39788 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39788 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39810 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39810 2024-01-21T04:58:34.093 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T04:58:34.094 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40882 2024-01-21T04:58:34.094 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40882 2024-01-21T04:58:43.003 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:10.910 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.008 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:10.916 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.044 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:10.951 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.065 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:10.940 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.068 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:10.943 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.079 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:10.986 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.083 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:10.990 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.091 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:10.966 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.105 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:11.012 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.821 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:11.728 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.822 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:11.729 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.823 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:11.698 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.824 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:11.700 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.830 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:11.737 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.832 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:11.707 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.871 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:11.778 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.872 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:11.779 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:43.880 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:11.788 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:48.019 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:15.925 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:48.020 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:15.927 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:48.032 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:15.939 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:48.069 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:15.944 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:48.070 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:15.976 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:48.072 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:15.979 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:48.072 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:15.947 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:48.083 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:15.958 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:58:48.083 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:15.990 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:03.008 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:30.915 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:03.061 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:30.968 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:03.061 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:30.937 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:03.065 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:30.940 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:03.065 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:30.972 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:03.067 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:30.974 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:03.081 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:30.957 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:03.082 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:30.989 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:03.084 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:30.991 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:08.066 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:35.973 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:08.070 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:35.977 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:08.082 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:35.989 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:08.113 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:36.020 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:08.116 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:35.992 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:08.118 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:36.025 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:08.118 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:35.993 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:08.128 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:36.004 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:08.128 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:36.035 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:23.175 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:51.082 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:23.178 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:51.085 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:23.179 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:51.087 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:23.197 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:51.104 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:23.225 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:51.101 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:23.227 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:51.102 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:23.233 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:51.141 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:23.240 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:51.115 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:23.247 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:51.155 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:28.125 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:56.032 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:28.126 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:56.033 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:28.139 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:26:56.046 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:28.174 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:56.049 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:28.177 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:56.052 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:28.186 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:26:56.062 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:28.187 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:56.094 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:28.190 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:56.097 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:28.201 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:26:56.108 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:43.109 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:11.016 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:43.159 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:11.066 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:43.189 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:11.097 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:43.208 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:11.083 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:43.210 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:11.117 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:43.211 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:11.087 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:43.212 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:11.120 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:43.226 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:11.102 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:43.228 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:11.136 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:48.171 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:16.078 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:48.173 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:16.080 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:48.183 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:16.090 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:48.215 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:16.090 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:48.218 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:16.093 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:48.221 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:16.129 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:48.223 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:16.130 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:48.228 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:16.103 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T04:59:48.233 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:16.140 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:03.214 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:31.121 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:03.218 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:31.126 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:03.234 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:31.141 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:03.264 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:31.139 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:03.266 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:31.174 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:03.269 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:31.145 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:03.270 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:31.177 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:03.283 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:31.159 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:03.285 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:31.192 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:08.223 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:36.130 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:08.227 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:36.134 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:08.239 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:36.147 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:08.274 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:36.181 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:08.276 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:36.152 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:08.276 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:36.184 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:08.280 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:36.155 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:08.288 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:36.195 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:08.290 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:36.165 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:23.261 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:51.168 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:23.263 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:51.170 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:23.279 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:51.186 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:23.309 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:51.184 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:23.311 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:51.218 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:23.311 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:51.187 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:23.315 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:51.222 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:23.326 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:51.201 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:23.329 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:51.236 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:28.230 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:56.137 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:28.278 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:56.185 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:28.290 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:27:56.197 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:28.323 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:56.199 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:28.326 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:56.201 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:28.336 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:27:56.212 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:28.346 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:56.253 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:28.351 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:56.258 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:28.362 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:27:56.269 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:43.578 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:11.485 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:43.581 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:11.488 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:43.606 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:11.513 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:43.625 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:11.501 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:43.627 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:11.534 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:43.628 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:11.503 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:43.630 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:11.537 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:43.652 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:11.527 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:43.654 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:11.561 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:48.448 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:16.355 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:48.451 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:16.358 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:48.463 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:16.370 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:48.497 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:16.405 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:48.498 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:16.373 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:48.500 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:16.375 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:48.501 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:16.408 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:48.510 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:16.386 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:00:48.513 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:16.420 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:03.448 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:31.354 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:03.496 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:31.403 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:03.511 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:31.418 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:03.543 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:31.450 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:03.544 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:31.419 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:03.545 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:31.420 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:03.546 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:31.453 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:03.559 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:31.434 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:03.561 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:31.469 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:08.506 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:36.413 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:08.510 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:36.417 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:08.522 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:36.429 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:08.559 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:36.434 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:08.560 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:36.436 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:08.572 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:36.447 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:08.575 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:36.483 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:08.581 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:36.488 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:08.591 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:36.498 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:14.656 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:42.563 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:14.658 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:42.565 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:14.673 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:42.580 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:14.704 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:42.579 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:14.708 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:42.583 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:14.710 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:42.617 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:14.712 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:42.619 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:14.724 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:42.599 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:14.726 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:42.634 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:23.539 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:51.446 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:23.544 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:51.451 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:23.555 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:51.462 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:23.588 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:51.463 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:23.590 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:51.497 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:23.591 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:51.467 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:23.592 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:51.499 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:23.602 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:51.477 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:23.604 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:51.511 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:28.555 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:56.462 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:28.559 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:56.467 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:28.571 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:28:56.478 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:28.605 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:56.480 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:28.605 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:56.512 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:28.606 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:56.481 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:28.607 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:56.514 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:28.617 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:28:56.492 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:28.618 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:28:56.525 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:43.597 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:11.504 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:43.601 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:11.508 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:43.617 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:11.524 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:43.648 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:11.523 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:43.650 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:11.525 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:43.651 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:11.558 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:43.652 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:11.559 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:43.664 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:11.539 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:43.667 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:11.574 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:48.608 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:16.515 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:48.613 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:16.520 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:48.625 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:16.532 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:48.659 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:16.566 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:48.661 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:16.536 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:48.661 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:16.569 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:48.664 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:16.539 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:48.673 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:16.580 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:01:48.675 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:16.550 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:03.644 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:31.550 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:03.647 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:31.554 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:03.667 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:31.574 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:03.699 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:31.606 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:03.699 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:31.575 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:03.701 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:31.576 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:03.702 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:31.609 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:03.714 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:31.589 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:03.717 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:31.624 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:08.660 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:36.566 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:08.662 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:36.569 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:08.671 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:36.578 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:08.702 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:36.577 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:08.704 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:36.612 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:08.705 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:36.580 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:08.708 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:36.615 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:08.716 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:36.591 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:08.720 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:36.628 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:23.691 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:51.598 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:23.695 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:51.602 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:23.710 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:51.617 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:23.744 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:51.651 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:23.745 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:51.619 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:23.745 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:51.652 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:23.747 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:51.623 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:23.759 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:51.666 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:23.765 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:51.640 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:28.708 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:56.615 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:28.712 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:56.619 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:28.724 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:29:56.632 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:28.757 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:56.632 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:28.758 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:56.634 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:28.759 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:56.666 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:28.761 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:56.668 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:28.769 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:29:56.644 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:28.772 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:29:56.680 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:43.757 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:11.664 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:43.762 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:11.669 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:43.777 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:11.684 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:43.808 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:11.715 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:43.810 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:11.685 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:43.812 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:11.720 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:43.813 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:11.689 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:43.831 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:11.738 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:43.834 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:11.709 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:48.755 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:16.662 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:48.759 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:16.666 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:48.772 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:16.679 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:48.805 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:16.680 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:48.807 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:16.682 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:48.808 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:16.715 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:48.809 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:16.716 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:48.818 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:16.694 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:02:48.820 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:16.727 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:03.794 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:31.701 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:03.799 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:31.706 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:03.816 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:31.723 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:03.853 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:31.759 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:03.855 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:31.762 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:03.864 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:31.739 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:03.867 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:31.743 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:03.876 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:31.783 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:03.881 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:31.756 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:08.762 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:36.669 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:08.810 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:36.717 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:08.827 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:36.734 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:08.857 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:36.732 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:08.858 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:36.765 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:08.858 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:36.734 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:08.862 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:36.769 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:08.869 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:36.745 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:08.874 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:36.782 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:23.845 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:51.752 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:23.847 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:51.754 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:23.869 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:51.776 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:23.899 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:51.774 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:23.902 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:51.778 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:23.903 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:51.810 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:23.905 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:51.813 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:23.918 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:51.793 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:23.921 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:51.828 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:28.807 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:56.714 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:28.853 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:56.729 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:28.855 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:56.762 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:28.856 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:56.731 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:28.868 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:30:56.744 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:28.870 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:30:56.778 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:28.904 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:56.811 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:28.905 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:56.812 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:28.916 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:30:56.823 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:43.851 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:11.757 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:43.900 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:11.807 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:43.902 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:11.778 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:43.905 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:11.780 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:43.925 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:11.831 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:43.925 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:11.799 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:43.946 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:11.853 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:43.948 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:11.856 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:43.968 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:11.876 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:48.907 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:16.814 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:48.911 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:16.818 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:48.925 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:16.832 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:48.955 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:16.863 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:48.960 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:16.867 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:48.961 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:16.836 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:48.963 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:16.839 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:48.972 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:16.879 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:03:48.976 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:16.851 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:03.947 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:31.854 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:03.949 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:31.856 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:03.965 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:31.872 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:03.999 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:31.874 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:03.999 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:31.907 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:04.002 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:31.877 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:04.002 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:31.909 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:04.020 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:31.895 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:04.021 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:31.929 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:08.913 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:36.820 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:08.960 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:36.867 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:08.975 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:36.882 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:09.010 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:36.917 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:09.011 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:36.886 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:09.013 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:36.888 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:09.013 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:36.920 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:09.024 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:36.931 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:09.024 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:36.900 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:23.996 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:51.902 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:23.999 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:51.906 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:24.014 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:51.921 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:24.047 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:51.954 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:24.048 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:51.923 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:24.049 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:51.924 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:24.050 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:51.957 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:24.064 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:51.939 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:24.066 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:51.973 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:26.386 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:54.293 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:26.388 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:54.295 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:26.408 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:54.315 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:26.436 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:54.312 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:26.440 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:54.316 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:26.452 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:54.359 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:26.453 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:54.329 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:26.454 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:54.361 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:26.467 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:54.374 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:29.012 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:56.919 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:29.014 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:56.921 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:29.022 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:31:56.929 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:29.056 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:56.931 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:29.061 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:56.936 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:29.062 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:56.969 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:29.063 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:56.971 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:29.071 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:31:56.946 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:29.074 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:31:56.981 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:44.045 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:11.952 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:44.049 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:11.957 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:44.065 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:11.972 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:44.100 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:12.007 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:44.101 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:11.976 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:44.101 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:12.008 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:44.104 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:11.980 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:44.116 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:12.023 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:44.119 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:11.995 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:49.055 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:16.962 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:49.058 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:16.965 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:49.072 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:16.979 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:49.105 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:16.980 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:49.106 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:16.981 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:49.107 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:17.014 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:49.111 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:17.018 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:49.115 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:16.991 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:49.122 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:17.029 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:59.131 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:27.038 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:59.135 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:27.042 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:59.163 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:27.070 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:59.182 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:27.057 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:59.185 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:27.061 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:59.194 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:27.101 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:59.199 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:27.106 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:59.203 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:27.078 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:04:59.218 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:27.125 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.138 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:32.045 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.141 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:32.048 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.154 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:32.061 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.186 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:32.061 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.187 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:32.062 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.190 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:32.097 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.192 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:32.099 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.200 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:32.076 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.207 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:32.114 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.894 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:32.769 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.895 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:32.771 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.910 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:32.785 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.980 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:32.887 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.982 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:32.889 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:04.998 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:32.905 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:05.056 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:32.963 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:05.062 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:32.970 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:05.097 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:33.004 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:19.369 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:47.276 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:19.374 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:47.281 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:19.428 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:47.335 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:19.439 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:47.314 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:19.441 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:47.348 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:19.442 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:47.350 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:19.449 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:47.324 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:19.473 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:47.380 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:19.483 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:47.358 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:24.216 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:52.123 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:24.221 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:52.129 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:24.241 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:32:52.149 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:24.273 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:52.148 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:24.274 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:52.150 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:24.282 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:52.189 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:24.283 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:52.191 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:24.289 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:32:52.165 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:24.297 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:32:52.204 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:25.987 INFO:tasks.workunit.client.0.smithi069.stdout: misc: Running main() from gmock_main.cc 2024-01-21T05:05:25.987 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-01-21T05:05:25.987 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [----------] Global test environment set-up. 2024-01-21T05:05:25.987 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-01-21T05:05:25.987 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.Version (4397 ms) 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4223 ms) 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.LongName (1802917 ms) 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3443 ms) 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3246 ms) 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4501 ms) 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-01-21T05:05:25.988 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.Exec (3589 ms) 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3374 ms) 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3057 ms) 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3585 ms) 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2222 ms) 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3494 ms) 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [----------] 12 tests from NeoRadosMisc (1842048 ms total) 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [----------] Global test environment tear-down 2024-01-21T05:05:25.989 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [==========] 12 tests from 1 test suite ran. (1842048 ms total) 2024-01-21T05:05:25.990 INFO:tasks.workunit.client.0.smithi069.stdout: misc: [ PASSED ] 12 tests. 2024-01-21T05:05:25.994 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.994 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40073 2024-01-21T05:05:25.994 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40073 2024-01-21T05:05:25.994 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.994 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40923 2024-01-21T05:05:25.994 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40923 2024-01-21T05:05:25.994 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40162 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40162 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40668 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40668 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40952 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40952 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40101 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40101 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40547 2024-01-21T05:05:25.995 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40547 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40811 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40811 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40630 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40630 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40907 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40907 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40411 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40411 2024-01-21T05:05:25.996 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39995 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39995 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40939 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40939 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40318 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40318 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39781 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39781 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39842 2024-01-21T05:05:25.997 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39842 2024-01-21T05:05:25.998 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.998 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40229 2024-01-21T05:05:25.998 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40229 2024-01-21T05:05:25.998 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.998 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=39886 2024-01-21T05:05:25.998 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 39886 2024-01-21T05:05:25.998 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.999 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40349 2024-01-21T05:05:25.999 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40349 2024-01-21T05:05:25.999 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.999 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40399 2024-01-21T05:05:25.999 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40399 2024-01-21T05:05:25.999 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.999 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40770 2024-01-21T05:05:25.999 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40770 2024-01-21T05:05:25.999 INFO:tasks.workunit.client.0.smithi069.stderr:+ for t in "${!pids[@]}" 2024-01-21T05:05:25.999 INFO:tasks.workunit.client.0.smithi069.stderr:+ pid=40967 2024-01-21T05:05:25.999 INFO:tasks.workunit.client.0.smithi069.stderr:+ wait 40967 2024-01-21T05:05:34.306 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:02.213 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:34.310 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:02.217 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:34.327 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:02.234 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:34.359 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:02.266 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:34.360 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:02.236 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:34.362 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:02.237 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:34.363 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:02.270 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:34.376 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:02.251 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:34.379 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:02.286 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:43.788 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: Running main() from gmock_main.cc 2024-01-21T05:05:43.789 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-01-21T05:05:43.789 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [----------] Global test environment set-up. 2024-01-21T05:05:43.789 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-01-21T05:05:43.789 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-01-21T05:05:43.789 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (14265 ms) 2024-01-21T05:05:43.789 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-01-21T05:05:43.789 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (1802936 ms) 2024-01-21T05:05:43.789 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-01-21T05:05:43.789 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3493 ms) 2024-01-21T05:05:43.789 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-01-21T05:05:43.789 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3228 ms) 2024-01-21T05:05:43.790 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-01-21T05:05:43.790 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3546 ms) 2024-01-21T05:05:43.790 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-01-21T05:05:43.790 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3289 ms) 2024-01-21T05:05:43.790 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-01-21T05:05:43.790 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (29091 ms) 2024-01-21T05:05:43.790 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (1859848 ms total) 2024-01-21T05:05:43.790 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: 2024-01-21T05:05:43.790 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [----------] Global test environment tear-down 2024-01-21T05:05:43.790 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (1859848 ms total) 2024-01-21T05:05:43.790 INFO:tasks.workunit.client.0.smithi069.stdout: write_operations: [ PASSED ] 7 tests. 2024-01-21T05:05:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ exit 0 2024-01-21T05:05:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ cleanup 2024-01-21T05:05:43.791 INFO:tasks.workunit.client.0.smithi069.stderr:+ pkill -P 39776 2024-01-21T05:05:43.881 INFO:tasks.workunit.client.0.smithi069.stderr:+ true 2024-01-21T05:05:43.882 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-01-21T05:05:43.882 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-01-21T05:05:43.913 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-01-21T05:05:43.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-01-21T05:05:44.242 DEBUG:teuthology.parallel:result is None 2024-01-21T05:05:44.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-01-21T05:05:44.269 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-01-21T05:05:44.269 DEBUG:teuthology.orchestra.run.smithi069:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-01-21T05:05:44.330 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-01-21T05:05:44.330 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-21T05:05:44.339 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-21T05:05:44.339 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T05:05:44.459 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:12.366 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:44.461 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:12.368 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:44.475 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:12.382 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:44.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.484+0000 7f8113d7c640 10 monclient: get_monmap_and_config 2024-01-21T05:05:44.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.484+0000 7f8113d7c640 10 monclient: build_initial_monmap 2024-01-21T05:05:44.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.484+0000 7f8113d7c640 10 monclient: monmap: 2024-01-21T05:05:44.485 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:44.485 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:44.485 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:44.485662+0000 2024-01-21T05:05:44.485 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:44.485662+0000 2024-01-21T05:05:44.485 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:44.485 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:44.486 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:44.486 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:44.486 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:44.486 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:44.486 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:44.486 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:44.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.485+0000 7f8113d7c640 1 Processor -- start 2024-01-21T05:05:44.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.485+0000 7f8113d7c640 1 -- start start 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.485+0000 7f8113d7c640 10 monclient: init 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f810c0a8a70 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f810c0a8e50 0x7f810c142450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 10 monclient(hunting): picked mon.noname-c con 0x7f810c0a8e50 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f810c13f6d0 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810c0aa690 con 0x7f810c13f6d0 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810c0aa970 con 0x7f810c0a8a70 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810c0a9f90 con 0x7f810c0a8e50 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:44.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.486+0000 7f8113d7c640 10 monclient(hunting): authenticate will time out at 3302.827816s 2024-01-21T05:05:44.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.489+0000 7f8112319640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f810c13f6d0 0x7f810c14c8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50166/0 (socket says 172.21.15.69:50166) 2024-01-21T05:05:44.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.489+0000 7f8112319640 1 -- 172.21.15.69:0/4085476729 learned_addr learned my addr 172.21.15.69:0/4085476729 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T05:05:44.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.499+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409277527 0 0) 0x7f810c0aa690 con 0x7f810c13f6d0 2024-01-21T05:05:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.499+0000 7f8110b16640 10 monclient(hunting): _negotiate 2024-01-21T05:05:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.499+0000 7f8110b16640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.499+0000 7f8110b16640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.499+0000 7f8110b16640 10 monclient(hunting): my global_id is 17412 2024-01-21T05:05:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.499+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f8003520 con 0x7f810c13f6d0 2024-01-21T05:05:44.510 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:12.417 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:44.511 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:12.419 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:44.523 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:12.398 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:44.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.523+0000 7f8111317640 1 --2- 172.21.15.69:0/4085476729 >> v2:172.21.15.144:3300/0 conn(0x7f810c0a8e50 0x7f810c142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:44.524 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:12.430 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:44.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.524+0000 7f8111317640 10 monclient(hunting): get_auth_request con 0x7f810c0a8e50 auth_method 0 2024-01-21T05:05:44.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.524+0000 7f8111317640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:44.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.524+0000 7f8111317640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:44.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.524+0000 7f8111317640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:44.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.525+0000 7f8111317640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:44.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.525+0000 7f8111317640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:44.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.525+0000 7f8111317640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:44.526 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:12.401 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:44.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3339256804 0 0) 0x7f80f8003520 con 0x7f810c13f6d0 2024-01-21T05:05:44.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 >> v2:172.21.15.144:3300/0 conn(0x7f810c0a8e50 msgr2=0x7f810c142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:44.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 --2- 172.21.15.69:0/4085476729 >> v2:172.21.15.144:3300/0 conn(0x7f810c0a8e50 0x7f810c142450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:44.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 >> v1:172.21.15.144:6789/0 conn(0x7f810c0a8a70 legacy=0x7f810c13efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient: _finish_hunting 0 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 monclient: found mon.noname-a-legacy 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.69:6789/0 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c0aac50 con 0x7f810c13f6d0 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient: _finish_auth 0 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient: _check_auth_tickets 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 125097436 0 0) 0x7f81040019f0 con 0x7f810c13f6d0 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient: dump: 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:44.533 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] but i'm connected to v1:172.21.15.69:6789/0, reconnecting 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 >> v1:172.21.15.69:6789/0 conn(0x7f810c13f6d0 legacy=0x7f810c14c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 --2- 172.21.15.69:0/4085476729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f80039c0 0x7f80f8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient(hunting): picked mon.a con 0x7f80f80039c0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 --2- 172.21.15.69:0/4085476729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f80047c0 0x7f80f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient(hunting): picked mon.c con 0x7f80f80047c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:44.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 --2- 172.21.15.69:0/4085476729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80f80051d0 0x7f80f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:44.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient(hunting): picked mon.b con 0x7f80f80051d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:44.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:44.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c0a9f90 con 0x7f80f80039c0 2024-01-21T05:05:44.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:44.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f800c120 con 0x7f80f80051d0 2024-01-21T05:05:44.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:44.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f800c2e0 con 0x7f80f80047c0 2024-01-21T05:05:44.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:44.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.531+0000 7f8110b16640 10 monclient(hunting): _finish_auth 0 2024-01-21T05:05:44.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.534+0000 7f8111317640 1 --2- 172.21.15.69:0/4085476729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f80047c0 0x7f80f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:44.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.536+0000 7f8111317640 10 monclient(hunting): get_auth_request con 0x7f80f80047c0 auth_method 0 2024-01-21T05:05:44.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.536+0000 7f8111317640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:44.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.536+0000 7f8111317640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:44.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.536+0000 7f8111317640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T05:05:44.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.538+0000 7f8111317640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:44.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.538+0000 7f8111317640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:44.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.538+0000 7f8111317640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T05:05:44.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.538+0000 7f8111b18640 1 --2- 172.21.15.69:0/4085476729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f80039c0 0x7f80f8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:44.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 10 monclient(hunting): handle_auth_done global_id 17412 payload 293 2024-01-21T05:05:44.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 1 -- 172.21.15.69:0/4085476729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80f80051d0 msgr2=0x7f80f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:44.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 1 --2- 172.21.15.69:0/4085476729 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f80f80051d0 0x7f80f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 1 -- 172.21.15.69:0/4085476729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f80039c0 msgr2=0x7f80f8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 1 --2- 172.21.15.69:0/4085476729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f80039c0 0x7f80f8003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 10 monclient: _finish_hunting 0 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 1 monclient: found mon.c 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8112319640 1 -- 172.21.15.69:0/4085476729 reap_dead start 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 1 -- 172.21.15.69:0/4085476729 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f8003520 con 0x7f80f80047c0 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 10 monclient: _finish_auth 0 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 10 monclient: _check_auth_tickets 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8111317640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8113d7c640 5 monclient: authenticate success, global_id 17412 2024-01-21T05:05:44.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.540+0000 7f8113d7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:44.547 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:12.422 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:44.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.547+0000 7f8111317640 1 --2- 172.21.15.69:0/4085476729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f80047c0 0x7f80f8004c40 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7f810c0a9250 tx=0x7f80f80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29aa75c7c58bd034 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:44.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc02c660 con 0x7f80f80047c0 2024-01-21T05:05:44.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8110b16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:44.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8110b16640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T05:05:44.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8110b16640 10 monclient: dump: 2024-01-21T05:05:44.566 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:44.566 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:44.566 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:44.566 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc02c7c0 con 0x7f80f80047c0 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8110b16640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8110b16640 1 -- 172.21.15.69:0/4085476729 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc02cb50 con 0x7f80f80047c0 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8110b16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8110b16640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8110b16640 10 monclient: dump: 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:44.567 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:44.568 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:44.568 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:44.568 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:44.568 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:44.568 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:44.568 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:44.568 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:44.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8113d7c640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:44.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.565+0000 7f8113d7c640 4 set_mon_vals no callback set 2024-01-21T05:05:44.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.566+0000 7f8113d7c640 10 monclient: shutdown 2024-01-21T05:05:44.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.566+0000 7f8113d7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:44.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.566+0000 7f8113d7c640 1 -- 172.21.15.69:0/4085476729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f80047c0 msgr2=0x7f80f8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:44.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.566+0000 7f8113d7c640 1 --2- 172.21.15.69:0/4085476729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f80047c0 0x7f80f8004c40 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7f810c0a9250 tx=0x7f80f80057f0 comp rx=0 tx=0).stop 2024-01-21T05:05:44.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.566+0000 7f8113d7c640 1 -- 172.21.15.69:0/4085476729 shutdown_connections 2024-01-21T05:05:44.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.566+0000 7f8113d7c640 1 --2- 172.21.15.69:0/4085476729 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f80f80047c0 0x7f80f8004c40 unknown :-1 s=CLOSED pgs=3321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.566+0000 7f8113d7c640 1 -- 172.21.15.69:0/4085476729 >> 172.21.15.69:0/4085476729 conn(0x7f810c066f00 msgr2=0x7f810c0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.566+0000 7f8113d7c640 1 -- 172.21.15.69:0/4085476729 shutdown_connections 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.566+0000 7f8113d7c640 1 -- 172.21.15.69:0/4085476729 wait complete. 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.567+0000 7f8113d7c640 10 monclient: build_initial_monmap 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.567+0000 7f8113d7c640 10 monclient: monmap: 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:44.572 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 librados: starting msgr at 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 librados: starting objecter 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 Processor -- start 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 -- start start 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 librados: setting wanted keys 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 librados: calling monclient init 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 10 monclient: init 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f80047c0 0x7f810c1da830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 10 monclient(hunting): picked mon.noname-a con 0x7f80f80047c0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:44.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f810c1db250 0x7f810c0ac390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:44.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 10 monclient(hunting): picked mon.noname-b con 0x7f810c1db250 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:44.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f810c0ac8d0 0x7f810c1d7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:44.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 10 monclient(hunting): picked mon.noname-c con 0x7f810c0ac8d0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:44.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:44.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c0adea0 con 0x7f80f80047c0 2024-01-21T05:05:44.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:44.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c0ae110 con 0x7f810c1db250 2024-01-21T05:05:44.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:44.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810c0abe70 con 0x7f810c0ac8d0 2024-01-21T05:05:44.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:44.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.568+0000 7f8113d7c640 10 monclient(hunting): authenticate will time out at 3302.910199s 2024-01-21T05:05:44.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.571+0000 7f8112319640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f810c0ac8d0 0x7f810c1d7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:44.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.572+0000 7f8112319640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f810c0ac8d0 0x7f810c1d7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:48476/0 (socket says 172.21.15.69:48476) 2024-01-21T05:05:44.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.572+0000 7f8112319640 1 -- 172.21.15.69:0/244671747 learned_addr learned my addr 172.21.15.69:0/244671747 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:44.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.572+0000 7f8112319640 10 monclient(hunting): get_auth_request con 0x7f810c0ac8d0 auth_method 0 2024-01-21T05:05:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.572+0000 7f8112319640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.572+0000 7f8112319640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.572+0000 7f8112319640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.573+0000 7f8112319640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.573+0000 7f8112319640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.573+0000 7f8112319640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.573+0000 7f8111b18640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f80047c0 0x7f810c1da830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.576+0000 7f8111b18640 10 monclient(hunting): get_auth_request con 0x7f80f80047c0 auth_method 0 2024-01-21T05:05:44.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.576+0000 7f8111b18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:44.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.576+0000 7f8111b18640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:44.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.576+0000 7f8111b18640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.577+0000 7f8112319640 10 monclient(hunting): handle_auth_done global_id 18806 payload 824 2024-01-21T05:05:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.577+0000 7f8112319640 1 -- 172.21.15.69:0/244671747 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f810c1db250 msgr2=0x7f810c0ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.577+0000 7f8112319640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f810c1db250 0x7f810c0ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.577+0000 7f8112319640 1 -- 172.21.15.69:0/244671747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f80047c0 msgr2=0x7f810c1da830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:44.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.577+0000 7f8112319640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f80047c0 0x7f810c1da830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:44.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.577+0000 7f8112319640 10 monclient: _finish_hunting 0 2024-01-21T05:05:44.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.577+0000 7f8112319640 1 monclient: found mon.noname-c 2024-01-21T05:05:44.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.577+0000 7f8112319640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:44.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.577+0000 7f8112319640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T05:05:44.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.577+0000 7f8112319640 1 -- 172.21.15.69:0/244671747 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c0abab0 con 0x7f810c0ac8d0 2024-01-21T05:05:44.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.582+0000 7f8112319640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f810c0ac8d0 0x7f810c1d7040 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f810c0b3940 tx=0x7f810400cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:44.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.582+0000 7f81027fc640 10 client.?.objecter ms_handle_connect 0x7f810c0ac8d0 2024-01-21T05:05:44.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.582+0000 7f81027fc640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:44.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.590+0000 7f81027fc640 1 -- 172.21.15.69:0/244671747 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810400ce50 con 0x7f810c0ac8d0 2024-01-21T05:05:44.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.590+0000 7f81027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:44.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.590+0000 7f81027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:05:44.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.590+0000 7f81027fc640 10 monclient: dump: 2024-01-21T05:05:44.592 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:44.592 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f81027fc640 10 monclient: _finish_auth 0 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f81027fc640 10 monclient: _check_auth_tickets 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f81027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f81027fc640 1 -- 172.21.15.69:0/244671747 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810400a5b0 con 0x7f810c0ac8d0 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f8113d7c640 5 monclient: authenticate success, global_id 18806 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f8113d7c640 10 monclient: _renew_subs 2024-01-21T05:05:44.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f8113d7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f8113d7c640 1 -- 172.21.15.69:0/244671747 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810c0abc40 con 0x7f810c0ac8d0 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f81027fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f81027fc640 1 -- 172.21.15.69:0/244671747 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810400a9a0 con 0x7f810c0ac8d0 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f81027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f81027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f81027fc640 10 monclient: dump: 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:44.594 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:44.595 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:44.595 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:44.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f8103fff640 4 set_mon_vals no callback set 2024-01-21T05:05:44.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f8113d7c640 10 client.18806.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:44.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f8113d7c640 10 monclient: _renew_subs 2024-01-21T05:05:44.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f8113d7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T05:05:44.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f8113d7c640 1 -- 172.21.15.69:0/244671747 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810c0acf30 con 0x7f810c0ac8d0 2024-01-21T05:05:44.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.591+0000 7f8113d7c640 1 librados: init done 2024-01-21T05:05:44.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.592+0000 7f8113d7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:44.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.592+0000 7f8113d7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:44.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.592+0000 7f8113d7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T05:05:44.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.592+0000 7f8113d7c640 1 -- 172.21.15.69:0/244671747 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810c04e970 con 0x7f810c0ac8d0 2024-01-21T05:05:44.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.913+0000 7f81027fc640 1 -- 172.21.15.69:0/244671747 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f810400ab00 con 0x7f810c0ac8d0 2024-01-21T05:05:44.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.913+0000 7f81027fc640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f80f006c860 0x7f80f006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:44.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.914+0000 7f8111b18640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f80f006c860 0x7f80f006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:44.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.914+0000 7f8111b18640 10 monclient: get_auth_request con 0x7f80f006c860 auth_method 0 2024-01-21T05:05:44.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.914+0000 7f8111b18640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f80f006c860 0x7f80f006ed20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8108001710 tx=0x7f8108009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:44.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.914+0000 7f81027fc640 10 client.18806.objecter ms_handle_connect 0x7f80f006c860 2024-01-21T05:05:44.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.939+0000 7f81027fc640 1 -- 172.21.15.69:0/244671747 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(1109..1109 src has 1032..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f81040584a0 con 0x7f810c0ac8d0 2024-01-21T05:05:44.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.939+0000 7f81027fc640 10 client.18806.objecter ms_dispatch 0x7f810c000b90 osd_map(1109..1109 src has 1032..1109) v4 2024-01-21T05:05:44.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.939+0000 7f81027fc640 3 client.18806.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:44.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.939+0000 7f81027fc640 3 client.18806.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:44.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.939+0000 7f81027fc640 20 client.18806.objecter dump_active .. 0 homeless 2024-01-21T05:05:44.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.939+0000 7f81027fc640 1 -- 172.21.15.69:0/244671747 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f8104057b70 con 0x7f810c0ac8d0 2024-01-21T05:05:44.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.939+0000 7f81027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:44.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:44.939+0000 7f81027fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:45.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.162+0000 7f8113d7c640 1 -- 172.21.15.69:0/244671747 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f810c13ef90 con 0x7f80f006c860 2024-01-21T05:05:45.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.169+0000 7f81027fc640 1 -- 172.21.15.69:0/244671747 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151767 (secure 0 0 0) 0x7f810c13ef90 con 0x7f80f006c860 2024-01-21T05:05:45.171 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T05:05:45.172 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-01-21T05:05:45.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 10 librados: watch_flush enter 2024-01-21T05:05:45.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 10 librados: watch_flush exit 2024-01-21T05:05:45.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 20 client.18806.objecter shutdown clearing up homeless session... 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 10 client.18806.objecter successfully canceled tick 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 -- 172.21.15.69:0/244671747 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f80f006c860 msgr2=0x7f80f006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f80f006c860 0x7f80f006ed20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8108001710 tx=0x7f8108009040 comp rx=0 tx=0).stop 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 10 monclient: shutdown 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 -- 172.21.15.69:0/244671747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f810c0ac8d0 msgr2=0x7f810c1d7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f810c0ac8d0 0x7f810c1d7040 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f810c0b3940 tx=0x7f810400cc00 comp rx=0 tx=0).stop 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 -- 172.21.15.69:0/244671747 shutdown_connections 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f80f006c860 0x7f80f006ed20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f810c0ac8d0 0x7f810c1d7040 unknown :-1 s=CLOSED pgs=3322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f810c1db250 0x7f810c0ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 --2- 172.21.15.69:0/244671747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80f80047c0 0x7f810c1da830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 -- 172.21.15.69:0/244671747 >> 172.21.15.69:0/244671747 conn(0x7f810c066f00 msgr2=0x7f810c142aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 -- 172.21.15.69:0/244671747 shutdown_connections 2024-01-21T05:05:45.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 -- 172.21.15.69:0/244671747 wait complete. 2024-01-21T05:05:45.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.171+0000 7f8113d7c640 1 librados: shutdown 2024-01-21T05:05:45.184 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":2079,"stamp":"2024-01-21T05:05:44.360791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":609024,"kb_used_data":91648,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754365696,"statfs":{"total":773094113280,"available":772470472704,"internally_reserved":0,"allocated":93847552,"data_stored":51661426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":529714736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":-2,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.328529"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":1046,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.057494+0000","last_change":"2024-01-21T04:34:59.893360+0000","last_active":"2024-01-21T05:05:25.057494+0000","last_peered":"2024-01-21T05:05:25.057494+0000","last_clean":"2024-01-21T05:05:25.057494+0000","last_became_active":"2024-01-21T04:34:59.874210+0000","last_became_peered":"2024-01-21T04:34:59.874210+0000","last_unstale":"2024-01-21T05:05:25.057494+0000","last_undegraded":"2024-01-21T05:05:25.057494+0000","last_fullsized":"2024-01-21T05:05:25.057494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:03:39.790631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1063,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.109554+0000","last_change":"2024-01-21T04:35:19.549398+0000","last_active":"2024-01-21T05:05:25.109554+0000","last_peered":"2024-01-21T05:05:25.109554+0000","last_clean":"2024-01-21T05:05:25.109554+0000","last_became_active":"2024-01-21T04:35:19.542355+0000","last_became_peered":"2024-01-21T04:35:19.542355+0000","last_unstale":"2024-01-21T05:05:25.109554+0000","last_undegraded":"2024-01-21T05:05:25.109554+0000","last_fullsized":"2024-01-21T05:05:25.109554+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:17:21.163426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1085,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.257683+0000","last_change":"2024-01-21T04:35:21.254053+0000","last_active":"2024-01-21T05:05:25.257683+0000","last_peered":"2024-01-21T05:05:25.257683+0000","last_clean":"2024-01-21T05:05:25.257683+0000","last_became_active":"2024-01-21T04:35:21.234113+0000","last_became_peered":"2024-01-21T04:35:21.234113+0000","last_unstale":"2024-01-21T05:05:25.257683+0000","last_undegraded":"2024-01-21T05:05:25.257683+0000","last_fullsized":"2024-01-21T05:05:25.257683+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:52:02.299953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1050,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.013476+0000","last_change":"2024-01-21T04:35:23.469523+0000","last_active":"2024-01-21T05:05:25.013476+0000","last_peered":"2024-01-21T05:05:25.013476+0000","last_clean":"2024-01-21T05:05:25.013476+0000","last_became_active":"2024-01-21T04:35:23.463341+0000","last_became_peered":"2024-01-21T04:35:23.463341+0000","last_unstale":"2024-01-21T05:05:25.013476+0000","last_undegraded":"2024-01-21T05:05:25.013476+0000","last_fullsized":"2024-01-21T05:05:25.013476+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:40:34.277268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1050,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.952116+0000","last_change":"2024-01-21T04:35:16.948777+0000","last_active":"2024-01-21T05:05:24.952116+0000","last_peered":"2024-01-21T05:05:24.952116+0000","last_clean":"2024-01-21T05:05:24.952116+0000","last_became_active":"2024-01-21T04:35:16.942042+0000","last_became_peered":"2024-01-21T04:35:16.942042+0000","last_unstale":"2024-01-21T05:05:24.952116+0000","last_undegraded":"2024-01-21T05:05:24.952116+0000","last_fullsized":"2024-01-21T05:05:24.952116+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:15:32.933069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"116'7","reported_seq":1077,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.918924+0000","last_change":"2024-01-21T04:35:25.379657+0000","last_active":"2024-01-21T05:05:24.918924+0000","last_peered":"2024-01-21T05:05:24.918924+0000","last_clean":"2024-01-21T05:05:24.918924+0000","last_became_active":"2024-01-21T04:35:25.364561+0000","last_became_peered":"2024-01-21T04:35:25.364561+0000","last_unstale":"2024-01-21T05:05:24.918924+0000","last_undegraded":"2024-01-21T05:05:24.918924+0000","last_fullsized":"2024-01-21T05:05:24.918924+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:49:49.587513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1084,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.251316+0000","last_change":"2024-01-21T04:34:53.473895+0000","last_active":"2024-01-21T05:05:25.251316+0000","last_peered":"2024-01-21T05:05:25.251316+0000","last_clean":"2024-01-21T05:05:25.251316+0000","last_became_active":"2024-01-21T04:34:53.465163+0000","last_became_peered":"2024-01-21T04:34:53.465163+0000","last_unstale":"2024-01-21T05:05:25.251316+0000","last_undegraded":"2024-01-21T05:05:25.251316+0000","last_fullsized":"2024-01-21T05:05:25.251316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:37:17.441183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1048,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.938561+0000","last_change":"2024-01-21T04:35:02.604107+0000","last_active":"2024-01-21T05:05:24.938561+0000","last_peered":"2024-01-21T05:05:24.938561+0000","last_clean":"2024-01-21T05:05:24.938561+0000","last_became_active":"2024-01-21T04:35:02.595993+0000","last_became_peered":"2024-01-21T04:35:02.595993+0000","last_unstale":"2024-01-21T05:05:24.938561+0000","last_undegraded":"2024-01-21T05:05:24.938561+0000","last_fullsized":"2024-01-21T05:05:24.938561+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:05:40.627686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1084,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.237542+0000","last_change":"2024-01-21T04:35:21.226772+0000","last_active":"2024-01-21T05:05:25.237542+0000","last_peered":"2024-01-21T05:05:25.237542+0000","last_clean":"2024-01-21T05:05:25.237542+0000","last_became_active":"2024-01-21T04:35:21.218816+0000","last_became_peered":"2024-01-21T04:35:21.218816+0000","last_unstale":"2024-01-21T05:05:25.237542+0000","last_undegraded":"2024-01-21T05:05:25.237542+0000","last_fullsized":"2024-01-21T05:05:25.237542+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:07:02.731410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1048,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.037035+0000","last_change":"2024-01-21T04:34:53.062240+0000","last_active":"2024-01-21T05:05:25.037035+0000","last_peered":"2024-01-21T05:05:25.037035+0000","last_clean":"2024-01-21T05:05:25.037035+0000","last_became_active":"2024-01-21T04:34:53.056955+0000","last_became_peered":"2024-01-21T04:34:53.056955+0000","last_unstale":"2024-01-21T05:05:25.037035+0000","last_undegraded":"2024-01-21T05:05:25.037035+0000","last_fullsized":"2024-01-21T05:05:25.037035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:03:25.878354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1065,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.081092+0000","last_change":"2024-01-21T04:35:00.848090+0000","last_active":"2024-01-21T05:05:25.081092+0000","last_peered":"2024-01-21T05:05:25.081092+0000","last_clean":"2024-01-21T05:05:25.081092+0000","last_became_active":"2024-01-21T04:35:00.838638+0000","last_became_peered":"2024-01-21T04:35:00.838638+0000","last_unstale":"2024-01-21T05:05:25.081092+0000","last_undegraded":"2024-01-21T05:05:25.081092+0000","last_fullsized":"2024-01-21T05:05:25.081092+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:25:11.256572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1046,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.049467+0000","last_change":"2024-01-21T04:35:28.444122+0000","last_active":"2024-01-21T05:05:25.049467+0000","last_peered":"2024-01-21T05:05:25.049467+0000","last_clean":"2024-01-21T05:05:25.049467+0000","last_became_active":"2024-01-21T04:35:28.438809+0000","last_became_peered":"2024-01-21T04:35:28.438809+0000","last_unstale":"2024-01-21T05:05:25.049467+0000","last_undegraded":"2024-01-21T05:05:25.049467+0000","last_fullsized":"2024-01-21T05:05:25.049467+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:27:38.339736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1049,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.984819+0000","last_change":"2024-01-21T04:35:00.835300+0000","last_active":"2024-01-21T05:05:24.984819+0000","last_peered":"2024-01-21T05:05:24.984819+0000","last_clean":"2024-01-21T05:05:24.984819+0000","last_became_active":"2024-01-21T04:35:00.824479+0000","last_became_peered":"2024-01-21T04:35:00.824479+0000","last_unstale":"2024-01-21T05:05:24.984819+0000","last_undegraded":"2024-01-21T05:05:24.984819+0000","last_fullsized":"2024-01-21T05:05:24.984819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:11:26.198997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":1179,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.909621+0000","last_change":"2024-01-21T04:33:33.164364+0000","last_active":"2024-01-21T05:05:24.909621+0000","last_peered":"2024-01-21T05:05:24.909621+0000","last_clean":"2024-01-21T05:05:24.909621+0000","last_became_active":"2024-01-21T04:33:33.133732+0000","last_became_peered":"2024-01-21T04:33:33.133732+0000","last_unstale":"2024-01-21T05:05:24.909621+0000","last_undegraded":"2024-01-21T05:05:24.909621+0000","last_fullsized":"2024-01-21T05:05:24.909621+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:29.982915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:29.982915+0000","last_clean_scrub_stamp":"2024-01-21T04:33:29.982915+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:02:00.259080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1057,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.958852+0000","last_change":"2024-01-21T04:33:48.369523+0000","last_active":"2024-01-21T05:05:24.958852+0000","last_peered":"2024-01-21T05:05:24.958852+0000","last_clean":"2024-01-21T05:05:24.958852+0000","last_became_active":"2024-01-21T04:33:48.364387+0000","last_became_peered":"2024-01-21T04:33:48.364387+0000","last_unstale":"2024-01-21T05:05:24.958852+0000","last_undegraded":"2024-01-21T05:05:24.958852+0000","last_fullsized":"2024-01-21T05:05:24.958852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:06:07.142172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1046,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.060352+0000","last_change":"2024-01-21T04:34:54.460251+0000","last_active":"2024-01-21T05:05:25.060352+0000","last_peered":"2024-01-21T05:05:25.060352+0000","last_clean":"2024-01-21T05:05:25.060352+0000","last_became_active":"2024-01-21T04:34:54.452154+0000","last_became_peered":"2024-01-21T04:34:54.452154+0000","last_unstale":"2024-01-21T05:05:25.060352+0000","last_undegraded":"2024-01-21T05:05:25.060352+0000","last_fullsized":"2024-01-21T05:05:25.060352+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:03:09.023273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1057,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.888382+0000","last_change":"2024-01-21T04:33:47.762316+0000","last_active":"2024-01-21T05:05:24.888382+0000","last_peered":"2024-01-21T05:05:24.888382+0000","last_clean":"2024-01-21T05:05:24.888382+0000","last_became_active":"2024-01-21T04:33:47.734733+0000","last_became_peered":"2024-01-21T04:33:47.734733+0000","last_unstale":"2024-01-21T05:05:24.888382+0000","last_undegraded":"2024-01-21T05:05:24.888382+0000","last_fullsized":"2024-01-21T05:05:24.888382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:16:54.133000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1049,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.016911+0000","last_change":"2024-01-21T04:35:05.297742+0000","last_active":"2024-01-21T05:05:25.016911+0000","last_peered":"2024-01-21T05:05:25.016911+0000","last_clean":"2024-01-21T05:05:25.016911+0000","last_became_active":"2024-01-21T04:35:05.290888+0000","last_became_peered":"2024-01-21T04:35:05.290888+0000","last_unstale":"2024-01-21T05:05:25.016911+0000","last_undegraded":"2024-01-21T05:05:25.016911+0000","last_fullsized":"2024-01-21T05:05:25.016911+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:51:16.216402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1053,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.991808+0000","last_change":"2024-01-21T04:33:48.697481+0000","last_active":"2024-01-21T05:05:24.991808+0000","last_peered":"2024-01-21T05:05:24.991808+0000","last_clean":"2024-01-21T05:05:24.991808+0000","last_became_active":"2024-01-21T04:33:48.670864+0000","last_became_peered":"2024-01-21T04:33:48.670864+0000","last_unstale":"2024-01-21T05:05:24.991808+0000","last_undegraded":"2024-01-21T05:05:24.991808+0000","last_fullsized":"2024-01-21T05:05:24.991808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:00:07.393617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1046,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.013422+0000","last_change":"2024-01-21T04:35:07.348347+0000","last_active":"2024-01-21T05:05:25.013422+0000","last_peered":"2024-01-21T05:05:25.013422+0000","last_clean":"2024-01-21T05:05:25.013422+0000","last_became_active":"2024-01-21T04:35:07.340277+0000","last_became_peered":"2024-01-21T04:35:07.340277+0000","last_unstale":"2024-01-21T05:05:25.013422+0000","last_undegraded":"2024-01-21T05:05:25.013422+0000","last_fullsized":"2024-01-21T05:05:25.013422+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:33:05.933710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1059,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.943953+0000","last_change":"2024-01-21T04:37:13.894971+0000","last_active":"2024-01-21T05:05:24.943953+0000","last_peered":"2024-01-21T05:05:24.943953+0000","last_clean":"2024-01-21T05:05:24.943953+0000","last_became_active":"2024-01-21T04:33:48.015845+0000","last_became_peered":"2024-01-21T04:33:48.015845+0000","last_unstale":"2024-01-21T05:05:24.943953+0000","last_undegraded":"2024-01-21T05:05:24.943953+0000","last_fullsized":"2024-01-21T05:05:24.943953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:42:22.571268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1048,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.907668+0000","last_change":"2024-01-21T04:35:16.132218+0000","last_active":"2024-01-21T05:05:24.907668+0000","last_peered":"2024-01-21T05:05:24.907668+0000","last_clean":"2024-01-21T05:05:24.907668+0000","last_became_active":"2024-01-21T04:35:16.127034+0000","last_became_peered":"2024-01-21T04:35:16.127034+0000","last_unstale":"2024-01-21T05:05:24.907668+0000","last_undegraded":"2024-01-21T05:05:24.907668+0000","last_fullsized":"2024-01-21T05:05:24.907668+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:29:27.366404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1054,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.059062+0000","last_change":"2024-01-21T04:33:48.068776+0000","last_active":"2024-01-21T05:05:25.059062+0000","last_peered":"2024-01-21T05:05:25.059062+0000","last_clean":"2024-01-21T05:05:25.059062+0000","last_became_active":"2024-01-21T04:33:48.032152+0000","last_became_peered":"2024-01-21T04:33:48.032152+0000","last_unstale":"2024-01-21T05:05:25.059062+0000","last_undegraded":"2024-01-21T05:05:25.059062+0000","last_fullsized":"2024-01-21T05:05:25.059062+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:12:43.256934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1046,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.002979+0000","last_change":"2024-01-21T04:34:55.542633+0000","last_active":"2024-01-21T05:05:25.002979+0000","last_peered":"2024-01-21T05:05:25.002979+0000","last_clean":"2024-01-21T05:05:25.002979+0000","last_became_active":"2024-01-21T04:34:55.527971+0000","last_became_peered":"2024-01-21T04:34:55.527971+0000","last_unstale":"2024-01-21T05:05:25.002979+0000","last_undegraded":"2024-01-21T05:05:25.002979+0000","last_fullsized":"2024-01-21T05:05:25.002979+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:19:42.501825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1056,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.986390+0000","last_change":"2024-01-21T04:33:47.840628+0000","last_active":"2024-01-21T05:05:24.986390+0000","last_peered":"2024-01-21T05:05:24.986390+0000","last_clean":"2024-01-21T05:05:24.986390+0000","last_became_active":"2024-01-21T04:33:47.835639+0000","last_became_peered":"2024-01-21T04:33:47.835639+0000","last_unstale":"2024-01-21T05:05:24.986390+0000","last_undegraded":"2024-01-21T05:05:24.986390+0000","last_fullsized":"2024-01-21T05:05:24.986390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:55:33.768752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1047,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.977642+0000","last_change":"2024-01-21T04:35:26.902591+0000","last_active":"2024-01-21T05:05:24.977642+0000","last_peered":"2024-01-21T05:05:24.977642+0000","last_clean":"2024-01-21T05:05:24.977642+0000","last_became_active":"2024-01-21T04:35:26.889439+0000","last_became_peered":"2024-01-21T04:35:26.889439+0000","last_unstale":"2024-01-21T05:05:24.977642+0000","last_undegraded":"2024-01-21T05:05:24.977642+0000","last_fullsized":"2024-01-21T05:05:24.977642+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:36:16.061320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1052,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.954450+0000","last_change":"2024-01-21T04:33:48.112047+0000","last_active":"2024-01-21T05:05:24.954450+0000","last_peered":"2024-01-21T05:05:24.954450+0000","last_clean":"2024-01-21T05:05:24.954450+0000","last_became_active":"2024-01-21T04:33:48.080898+0000","last_became_peered":"2024-01-21T04:33:48.080898+0000","last_unstale":"2024-01-21T05:05:24.954450+0000","last_undegraded":"2024-01-21T05:05:24.954450+0000","last_fullsized":"2024-01-21T05:05:24.954450+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:54:29.756588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1064,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.082836+0000","last_change":"2024-01-21T04:35:36.906825+0000","last_active":"2024-01-21T05:05:25.082836+0000","last_peered":"2024-01-21T05:05:25.082836+0000","last_clean":"2024-01-21T05:05:25.082836+0000","last_became_active":"2024-01-21T04:35:36.901570+0000","last_became_peered":"2024-01-21T04:35:36.901570+0000","last_unstale":"2024-01-21T05:05:25.082836+0000","last_undegraded":"2024-01-21T05:05:25.082836+0000","last_fullsized":"2024-01-21T05:05:25.082836+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:17:42.698152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1055,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.015022+0000","last_change":"2024-01-21T04:33:48.051441+0000","last_active":"2024-01-21T05:05:25.015022+0000","last_peered":"2024-01-21T05:05:25.015022+0000","last_clean":"2024-01-21T05:05:25.015022+0000","last_became_active":"2024-01-21T04:33:48.046404+0000","last_became_peered":"2024-01-21T04:33:48.046404+0000","last_unstale":"2024-01-21T05:05:25.015022+0000","last_undegraded":"2024-01-21T05:05:25.015022+0000","last_fullsized":"2024-01-21T05:05:25.015022+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:19:24.446344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1047,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.998878+0000","last_change":"2024-01-21T04:35:19.621072+0000","last_active":"2024-01-21T05:05:24.998878+0000","last_peered":"2024-01-21T05:05:24.998878+0000","last_clean":"2024-01-21T05:05:24.998878+0000","last_became_active":"2024-01-21T04:35:19.611575+0000","last_became_peered":"2024-01-21T04:35:19.611575+0000","last_unstale":"2024-01-21T05:05:24.998878+0000","last_undegraded":"2024-01-21T05:05:24.998878+0000","last_fullsized":"2024-01-21T05:05:24.998878+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:09:33.751134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1082,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.249530+0000","last_change":"2024-01-21T04:34:53.506620+0000","last_active":"2024-01-21T05:05:25.249530+0000","last_peered":"2024-01-21T05:05:25.249530+0000","last_clean":"2024-01-21T05:05:25.249530+0000","last_became_active":"2024-01-21T04:34:53.495212+0000","last_became_peered":"2024-01-21T04:34:53.495212+0000","last_unstale":"2024-01-21T05:05:25.249530+0000","last_undegraded":"2024-01-21T05:05:25.249530+0000","last_fullsized":"2024-01-21T05:05:25.249530+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:38:49.385646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1045,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.990199+0000","last_change":"2024-01-21T04:34:58.579556+0000","last_active":"2024-01-21T05:05:24.990199+0000","last_peered":"2024-01-21T05:05:24.990199+0000","last_clean":"2024-01-21T05:05:24.990199+0000","last_became_active":"2024-01-21T04:34:58.492357+0000","last_became_peered":"2024-01-21T04:34:58.492357+0000","last_unstale":"2024-01-21T05:05:24.990199+0000","last_undegraded":"2024-01-21T05:05:24.990199+0000","last_fullsized":"2024-01-21T05:05:24.990199+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:55:43.736268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1045,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.979947+0000","last_change":"2024-01-21T04:35:26.240386+0000","last_active":"2024-01-21T05:05:24.979947+0000","last_peered":"2024-01-21T05:05:24.979947+0000","last_clean":"2024-01-21T05:05:24.979947+0000","last_became_active":"2024-01-21T04:35:25.928650+0000","last_became_peered":"2024-01-21T04:35:25.928650+0000","last_unstale":"2024-01-21T05:05:24.979947+0000","last_undegraded":"2024-01-21T05:05:24.979947+0000","last_fullsized":"2024-01-21T05:05:24.979947+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:38:53.872497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1062,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.970780+0000","last_change":"2024-01-21T04:35:00.395084+0000","last_active":"2024-01-21T05:05:24.970780+0000","last_peered":"2024-01-21T05:05:24.970780+0000","last_clean":"2024-01-21T05:05:24.970780+0000","last_became_active":"2024-01-21T04:35:00.387495+0000","last_became_peered":"2024-01-21T04:35:00.387495+0000","last_unstale":"2024-01-21T05:05:24.970780+0000","last_undegraded":"2024-01-21T05:05:24.970780+0000","last_fullsized":"2024-01-21T05:05:24.970780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:17:14.645461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1082,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.255370+0000","last_change":"2024-01-21T04:35:16.819308+0000","last_active":"2024-01-21T05:05:25.255370+0000","last_peered":"2024-01-21T05:05:25.255370+0000","last_clean":"2024-01-21T05:05:25.255370+0000","last_became_active":"2024-01-21T04:35:16.679256+0000","last_became_peered":"2024-01-21T04:35:16.679256+0000","last_unstale":"2024-01-21T05:05:25.255370+0000","last_undegraded":"2024-01-21T05:05:25.255370+0000","last_fullsized":"2024-01-21T05:05:25.255370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:48:12.462617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1062,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.959859+0000","last_change":"2024-01-21T04:35:25.009419+0000","last_active":"2024-01-21T05:05:24.959859+0000","last_peered":"2024-01-21T05:05:24.959859+0000","last_clean":"2024-01-21T05:05:24.959859+0000","last_became_active":"2024-01-21T04:35:25.004049+0000","last_became_peered":"2024-01-21T04:35:25.004049+0000","last_unstale":"2024-01-21T05:05:24.959859+0000","last_undegraded":"2024-01-21T05:05:24.959859+0000","last_fullsized":"2024-01-21T05:05:24.959859+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:44:01.746389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1048,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.015015+0000","last_change":"2024-01-21T04:35:23.577107+0000","last_active":"2024-01-21T05:05:25.015015+0000","last_peered":"2024-01-21T05:05:25.015015+0000","last_clean":"2024-01-21T05:05:25.015015+0000","last_became_active":"2024-01-21T04:35:23.567833+0000","last_became_peered":"2024-01-21T04:35:23.567833+0000","last_unstale":"2024-01-21T05:05:25.015015+0000","last_undegraded":"2024-01-21T05:05:25.015015+0000","last_fullsized":"2024-01-21T05:05:25.015015+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:50:25.014755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1047,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.021801+0000","last_change":"2024-01-21T04:34:57.160123+0000","last_active":"2024-01-21T05:05:25.021801+0000","last_peered":"2024-01-21T05:05:25.021801+0000","last_clean":"2024-01-21T05:05:25.021801+0000","last_became_active":"2024-01-21T04:34:57.153977+0000","last_became_peered":"2024-01-21T04:34:57.153977+0000","last_unstale":"2024-01-21T05:05:25.021801+0000","last_undegraded":"2024-01-21T05:05:25.021801+0000","last_fullsized":"2024-01-21T05:05:25.021801+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:18:26.237354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1047,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.906165+0000","last_change":"2024-01-21T04:35:28.601417+0000","last_active":"2024-01-21T05:05:24.906165+0000","last_peered":"2024-01-21T05:05:24.906165+0000","last_clean":"2024-01-21T05:05:24.906165+0000","last_became_active":"2024-01-21T04:35:28.583632+0000","last_became_peered":"2024-01-21T04:35:28.583632+0000","last_unstale":"2024-01-21T05:05:24.906165+0000","last_undegraded":"2024-01-21T05:05:24.906165+0000","last_fullsized":"2024-01-21T05:05:24.906165+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:35:42.169701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1082,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.225971+0000","last_change":"2024-01-21T04:35:16.742449+0000","last_active":"2024-01-21T05:05:25.225971+0000","last_peered":"2024-01-21T05:05:25.225971+0000","last_clean":"2024-01-21T05:05:25.225971+0000","last_became_active":"2024-01-21T04:35:16.734793+0000","last_became_peered":"2024-01-21T04:35:16.734793+0000","last_unstale":"2024-01-21T05:05:25.225971+0000","last_undegraded":"2024-01-21T05:05:25.225971+0000","last_fullsized":"2024-01-21T05:05:25.225971+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:19:56.812663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1064,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.059570+0000","last_change":"2024-01-21T04:35:14.711183+0000","last_active":"2024-01-21T05:05:25.059570+0000","last_peered":"2024-01-21T05:05:25.059570+0000","last_clean":"2024-01-21T05:05:25.059570+0000","last_became_active":"2024-01-21T04:35:14.685816+0000","last_became_peered":"2024-01-21T04:35:14.685816+0000","last_unstale":"2024-01-21T05:05:25.059570+0000","last_undegraded":"2024-01-21T05:05:25.059570+0000","last_fullsized":"2024-01-21T05:05:25.059570+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:04:28.368206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":21,"seq":90194313604,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74468,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297372,"statfs":{"total":96636764160,"available":96560508928,"internally_reserved":0,"allocated":11628544,"data_stored":6358216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:02 2024","interfaces":[{"interface":"back","average":{"1min":16.902000000000001,"5min":11.343,"15min":17.483000000000001},"min":{"1min":6.3620000000000001,"5min":5.4320000000000004,"15min":4.5259999999999998},"max":{"1min":130.69499999999999,"5min":130.69499999999999,"15min":305.86900000000003},"last":17.004999999999999},{"interface":"front","average":{"1min":19.975999999999999,"5min":14.539,"15min":20.905000000000001},"min":{"1min":10.542999999999999,"5min":8.0370000000000008,"15min":4.6710000000000003},"max":{"1min":121.011,"5min":121.011,"15min":281.79000000000002},"last":19.651}]},{"osd":1,"last update":"Sun Jan 21 05:05:28 2024","interfaces":[{"interface":"back","average":{"1min":30.696999999999999,"5min":16.850999999999999,"15min":19.510000000000002},"min":{"1min":9.0579999999999998,"5min":7.6550000000000002,"15min":6.1280000000000001},"max":{"1min":151.977,"5min":151.977,"15min":249.82599999999999},"last":20.242999999999999},{"interface":"front","average":{"1min":21.411999999999999,"5min":10.917,"15min":14.314},"min":{"1min":5.5709999999999997,"5min":4.8860000000000001,"15min":4.4429999999999996},"max":{"1min":133.238,"5min":133.238,"15min":185.57599999999999},"last":12.459}]},{"osd":2,"last update":"Sun Jan 21 05:05:16 2024","interfaces":[{"interface":"back","average":{"1min":18.077000000000002,"5min":11.068,"15min":16.349},"min":{"1min":6.9770000000000003,"5min":6.4000000000000004,"15min":6.4000000000000004},"max":{"1min":102.258,"5min":102.258,"15min":260.14999999999998},"last":13.958},{"interface":"front","average":{"1min":24.271000000000001,"5min":15.379,"15min":22.163},"min":{"1min":9.6739999999999995,"5min":7.8390000000000004,"15min":7.8390000000000004},"max":{"1min":108.218,"5min":108.218,"15min":261.67200000000003},"last":15.444000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:05:07 2024","interfaces":[{"interface":"back","average":{"1min":21.986999999999998,"5min":12.903,"15min":16.385999999999999},"min":{"1min":6.5780000000000003,"5min":6.1299999999999999,"15min":6.0830000000000002},"max":{"1min":173.92099999999999,"5min":173.92099999999999,"15min":179.381},"last":14.923},{"interface":"front","average":{"1min":22.701000000000001,"5min":15.462,"15min":18.68},"min":{"1min":11.153,"5min":7.0839999999999996,"15min":6.766},"max":{"1min":120.02800000000001,"5min":120.02800000000001,"15min":209.67099999999999},"last":17.731999999999999}]},{"osd":5,"last update":"Sun Jan 21 05:04:58 2024","interfaces":[{"interface":"back","average":{"1min":15.048,"5min":15.337,"15min":22.681999999999999},"min":{"1min":11.866,"5min":7.8719999999999999,"15min":6.9800000000000004},"max":{"1min":17.161999999999999,"5min":30.702000000000002,"15min":318.983},"last":22.001000000000001},{"interface":"front","average":{"1min":12.109,"5min":12.140000000000001,"15min":21.193999999999999},"min":{"1min":8.077,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.620999999999999,"5min":21.742000000000001,"15min":341.21699999999998},"last":18.527000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:05:03 2024","interfaces":[{"interface":"back","average":{"1min":23.292000000000002,"5min":17.651,"15min":26.388999999999999},"min":{"1min":13.853999999999999,"5min":10.154999999999999,"15min":7.1859999999999999},"max":{"1min":110.64100000000001,"5min":110.64100000000001,"15min":329.39100000000002},"last":22.591000000000001},{"interface":"front","average":{"1min":24.414000000000001,"5min":19.693999999999999,"15min":27.457000000000001},"min":{"1min":13.991,"5min":11.544,"15min":11.026},"max":{"1min":102.995,"5min":102.995,"15min":330.54899999999998},"last":24.09}]},{"osd":7,"last update":"Sun Jan 21 05:05:28 2024","interfaces":[{"interface":"back","average":{"1min":40.505000000000003,"5min":19.132000000000001,"15min":21.704000000000001},"min":{"1min":9.1530000000000005,"5min":7.8639999999999999,"15min":7.7240000000000002},"max":{"1min":233.38499999999999,"5min":233.38499999999999,"15min":233.38499999999999},"last":20.902999999999999},{"interface":"front","average":{"1min":41.484999999999999,"5min":21.462,"15min":23.84},"min":{"1min":9.9809999999999999,"5min":9.9120000000000008,"15min":9.9120000000000008},"max":{"1min":264.34800000000001,"5min":264.34800000000001,"15min":264.34800000000001},"last":23.416}]}]},{"osd":6,"up_from":20,"seq":85899346308,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74476,"kb_used_data":11364,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297364,"statfs":{"total":96636764160,"available":96560500736,"internally_reserved":0,"allocated":11636736,"data_stored":6358225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":7.9119999999999999,"5min":8.2769999999999992,"15min":18.212},"min":{"1min":4.3540000000000001,"5min":4.109,"15min":4.0830000000000002},"max":{"1min":13.827,"5min":18.007999999999999,"15min":293.416},"last":5.2009999999999996},{"interface":"front","average":{"1min":8.2260000000000009,"5min":8.4949999999999992,"15min":17.937999999999999},"min":{"1min":4.3849999999999998,"5min":4.3849999999999998,"15min":4.2220000000000004},"max":{"1min":11.989000000000001,"5min":19.242000000000001,"15min":258.678},"last":6.7649999999999997}]},{"osd":1,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":12.247,"5min":11.308999999999999,"15min":17.893999999999998},"min":{"1min":7.6929999999999996,"5min":6.2839999999999998,"15min":6.0650000000000004},"max":{"1min":18.242999999999999,"5min":19.047999999999998,"15min":267.21899999999999},"last":12.834},{"interface":"front","average":{"1min":15.005000000000001,"5min":14.324,"15min":23.260000000000002},"min":{"1min":10.436999999999999,"5min":8.6660000000000004,"15min":7.3339999999999996},"max":{"1min":18.088999999999999,"5min":20.760000000000002,"15min":271.875},"last":13.657}]},{"osd":2,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":11.577,"5min":11.223000000000001,"15min":19.917999999999999},"min":{"1min":6.0039999999999996,"5min":6.0039999999999996,"15min":6.0039999999999996},"max":{"1min":19.081,"5min":19.081,"15min":274.54000000000002},"last":16.366},{"interface":"front","average":{"1min":14.459,"5min":13.849,"15min":17.952000000000002},"min":{"1min":7.5869999999999997,"5min":7.5869999999999997,"15min":5.9420000000000002},"max":{"1min":20.527000000000001,"5min":20.527000000000001,"15min":163.05199999999999},"last":18.417000000000002}]},{"osd":3,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":12.615,"5min":12.771000000000001,"15min":18.033999999999999},"min":{"1min":6.7919999999999998,"5min":6.7919999999999998,"15min":6.7549999999999999},"max":{"1min":18.863,"5min":20.734999999999999,"15min":156.47},"last":8.1890000000000001},{"interface":"front","average":{"1min":15.964,"5min":15.805,"15min":22.997},"min":{"1min":10.096,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":19.196999999999999,"5min":20.001999999999999,"15min":275.81},"last":15.608000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":17.013000000000002,"5min":16.920999999999999,"15min":25.783000000000001},"min":{"1min":13.029999999999999,"5min":11.298,"15min":9.625},"max":{"1min":20.202999999999999,"5min":21.173999999999999,"15min":280.101},"last":17.108000000000001},{"interface":"front","average":{"1min":18.748000000000001,"5min":18.567,"15min":22.736999999999998},"min":{"1min":15.997999999999999,"5min":11.999000000000001,"15min":6.4119999999999999},"max":{"1min":21.719000000000001,"5min":22.102,"15min":273.45600000000002},"last":17.837}]},{"osd":5,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":11.468,"15min":21.411999999999999},"min":{"1min":6.5499999999999998,"5min":6.1699999999999999,"15min":6.1699999999999999},"max":{"1min":20.835000000000001,"5min":20.835000000000001,"15min":275.233},"last":9.5609999999999999},{"interface":"front","average":{"1min":14.619,"5min":14.329000000000001,"15min":22.456},"min":{"1min":7.9710000000000001,"5min":7.9710000000000001,"15min":6.9500000000000002},"max":{"1min":23.047999999999998,"5min":23.047999999999998,"15min":288.76499999999999},"last":10.563000000000001}]},{"osd":7,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":12.044,"5min":12.678000000000001,"15min":21.946000000000002},"min":{"1min":8.0860000000000003,"5min":7.5220000000000002,"15min":7.5220000000000002},"max":{"1min":19.792000000000002,"5min":20.399000000000001,"15min":366.68000000000001},"last":11.395},{"interface":"front","average":{"1min":15.031000000000001,"5min":15.369,"15min":22.943999999999999},"min":{"1min":10.625,"5min":9.6790000000000003,"15min":9.6790000000000003},"max":{"1min":20.709,"5min":21.266999999999999,"15min":281.78500000000003},"last":15.943}]}]},{"osd":4,"up_from":20,"seq":85899346308,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74472,"kb_used_data":11360,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297368,"statfs":{"total":96636764160,"available":96560504832,"internally_reserved":0,"allocated":11632640,"data_stored":6358220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":17.173999999999999,"5min":11.048999999999999,"15min":16.907},"min":{"1min":5.7629999999999999,"5min":5.7629999999999999,"15min":4.2409999999999997},"max":{"1min":154.04499999999999,"5min":154.04499999999999,"15min":294.92099999999999},"last":7.7859999999999996},{"interface":"front","average":{"1min":18.936,"5min":14.742000000000001,"15min":19.247},"min":{"1min":8.673,"5min":6.4379999999999997,"15min":4.6699999999999999},"max":{"1min":99.890000000000001,"5min":99.890000000000001,"15min":292.43700000000001},"last":27.164000000000001}]},{"osd":1,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":10.364000000000001,"5min":9.9369999999999994,"15min":15.395},"min":{"1min":6.0350000000000001,"5min":5.1669999999999998,"15min":4.1470000000000002},"max":{"1min":60.579999999999998,"5min":60.579999999999998,"15min":221.84999999999999},"last":21.420999999999999},{"interface":"front","average":{"1min":12.541,"5min":13.403,"15min":17.917000000000002},"min":{"1min":7.2190000000000003,"5min":7.032,"15min":6.3959999999999999},"max":{"1min":22.050000000000001,"5min":36.454999999999998,"15min":299.11599999999999},"last":28.007000000000001}]},{"osd":2,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":13.195,"5min":13.805999999999999,"15min":19.687999999999999},"min":{"1min":7.6289999999999996,"5min":7.0209999999999999,"15min":5.7439999999999998},"max":{"1min":23.762,"5min":30.693999999999999,"15min":293.60199999999998},"last":19.698},{"interface":"front","average":{"1min":13.038,"5min":11.211,"15min":20.07},"min":{"1min":7.532,"5min":6.4829999999999997,"15min":6.4829999999999997},"max":{"1min":20.181999999999999,"5min":34.270000000000003,"15min":303.94799999999998},"last":20.323}]},{"osd":3,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":29.370000000000001,"5min":16.681000000000001,"15min":22.192},"min":{"1min":6.0940000000000003,"5min":6.0940000000000003,"15min":6.0940000000000003},"max":{"1min":226.172,"5min":226.172,"15min":340.21100000000001},"last":25.408999999999999},{"interface":"front","average":{"1min":36.143999999999998,"5min":16.957999999999998,"15min":22.98},"min":{"1min":9.0310000000000006,"5min":6.6529999999999996,"15min":6.5579999999999998},"max":{"1min":211.38300000000001,"5min":211.38300000000001,"15min":302.72500000000002},"last":17.052}]},{"osd":5,"last update":"Sun Jan 21 05:04:58 2024","interfaces":[{"interface":"back","average":{"1min":23.199999999999999,"5min":15.675000000000001,"15min":23.965},"min":{"1min":8.4740000000000002,"5min":7.2510000000000003,"15min":7.2510000000000003},"max":{"1min":206.16800000000001,"5min":206.16800000000001,"15min":305.17200000000003},"last":24.681000000000001},{"interface":"front","average":{"1min":24.972999999999999,"5min":18.754999999999999,"15min":29.27},"min":{"1min":9.9359999999999999,"5min":9.9359999999999999,"15min":9.9359999999999999},"max":{"1min":192.066,"5min":192.066,"15min":326.19900000000001},"last":28.780000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":24.899999999999999,"5min":15.957000000000001,"15min":23.010999999999999},"min":{"1min":6.8769999999999998,"5min":6.7409999999999997,"15min":6.7409999999999997},"max":{"1min":204.32900000000001,"5min":204.32900000000001,"15min":248.09800000000001},"last":18.550999999999998},{"interface":"front","average":{"1min":27.047000000000001,"5min":18.641999999999999,"15min":26.821000000000002},"min":{"1min":7.7800000000000002,"5min":7.7800000000000002,"15min":7.7800000000000002},"max":{"1min":205.94800000000001,"5min":205.94800000000001,"15min":333.46899999999999},"last":22.148}]},{"osd":7,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":28.289999999999999,"5min":17.347000000000001,"15min":20.901},"min":{"1min":9.5459999999999994,"5min":7.3970000000000002,"15min":6.5430000000000001},"max":{"1min":260.63499999999999,"5min":260.63499999999999,"15min":312.41199999999998},"last":23.809000000000001},{"interface":"front","average":{"1min":29.556000000000001,"5min":19.172999999999998,"15min":25.035},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":236.80799999999999,"5min":236.80799999999999,"15min":247.44499999999999},"last":22.821999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74468,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297372,"statfs":{"total":96636764160,"available":96560508928,"internally_reserved":0,"allocated":11628544,"data_stored":6358216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:21 2024","interfaces":[{"interface":"back","average":{"1min":26.605,"5min":13.619999999999999,"15min":20.289000000000001},"min":{"1min":4.9279999999999999,"5min":4.46,"15min":3.847},"max":{"1min":244.893,"5min":244.893,"15min":272.279},"last":10.243},{"interface":"front","average":{"1min":27.978000000000002,"5min":12.045999999999999,"15min":19.513000000000002},"min":{"1min":4.516,"5min":4.0579999999999998,"15min":4.0579999999999998},"max":{"1min":224.22800000000001,"5min":224.22800000000001,"15min":356.99799999999999},"last":4.5419999999999998}]},{"osd":1,"last update":"Sun Jan 21 05:04:54 2024","interfaces":[{"interface":"back","average":{"1min":8.9190000000000005,"5min":8.7639999999999993,"15min":15.289999999999999},"min":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7759999999999998},"max":{"1min":19.866,"5min":19.866,"15min":276.94099999999997},"last":7.2990000000000004},{"interface":"front","average":{"1min":13.728,"5min":13.041,"15min":20.687999999999999},"min":{"1min":7.8600000000000003,"5min":7.4729999999999999,"15min":6.383},"max":{"1min":23.332000000000001,"5min":23.332000000000001,"15min":232.74299999999999},"last":10.858000000000001}]},{"osd":2,"last update":"Sun Jan 21 05:05:21 2024","interfaces":[{"interface":"back","average":{"1min":32.673999999999999,"5min":17.591000000000001,"15min":25.193000000000001},"min":{"1min":8.125,"5min":7.5919999999999996,"15min":7.5919999999999996},"max":{"1min":259.29899999999998,"5min":259.29899999999998,"15min":284.34100000000001},"last":11.44},{"interface":"front","average":{"1min":35.651000000000003,"5min":20.353000000000002,"15min":24.027999999999999},"min":{"1min":14.430999999999999,"5min":11.855,"15min":6.0229999999999997},"max":{"1min":260.87799999999999,"5min":260.87799999999999,"15min":260.87799999999999},"last":12.641}]},{"osd":3,"last update":"Sun Jan 21 05:05:37 2024","interfaces":[{"interface":"back","average":{"1min":41.347000000000001,"5min":21.199000000000002,"15min":26.056000000000001},"min":{"1min":8.8219999999999992,"5min":8.7919999999999998,"15min":7.8109999999999999},"max":{"1min":263.053,"5min":263.053,"15min":276.61399999999998},"last":12.050000000000001},{"interface":"front","average":{"1min":44.539999999999999,"5min":23.657,"15min":29.605},"min":{"1min":14.435,"5min":13.215,"15min":10.497},"max":{"1min":265.154,"5min":265.154,"15min":277.28100000000001},"last":15.707000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:05:37 2024","interfaces":[{"interface":"back","average":{"1min":50.813000000000002,"5min":18.835000000000001,"15min":17.899000000000001},"min":{"1min":7.5730000000000004,"5min":5.8620000000000001,"15min":5.8620000000000001},"max":{"1min":316.40699999999998,"5min":316.40699999999998,"15min":316.40699999999998},"last":8.3409999999999993},{"interface":"front","average":{"1min":45.965000000000003,"5min":23.863,"15min":25.629000000000001},"min":{"1min":15.303000000000001,"5min":12.269,"15min":9.1609999999999996},"max":{"1min":256.67099999999999,"5min":256.67099999999999,"15min":263.23099999999999},"last":16.457000000000001}]},{"osd":5,"last update":"Sun Jan 21 05:05:22 2024","interfaces":[{"interface":"back","average":{"1min":38.140999999999998,"5min":16.295000000000002,"15min":21.081},"min":{"1min":8.4489999999999998,"5min":6.6459999999999999,"15min":6.6429999999999998},"max":{"1min":223.56999999999999,"5min":223.56999999999999,"15min":223.56999999999999},"last":13.917999999999999},{"interface":"front","average":{"1min":40.212000000000003,"5min":19.149000000000001,"15min":23.727},"min":{"1min":8.6560000000000006,"5min":8.6240000000000006,"15min":8.5429999999999993},"max":{"1min":224.81100000000001,"5min":224.81100000000001,"15min":224.81100000000001},"last":14.672000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:04:54 2024","interfaces":[{"interface":"back","average":{"1min":13.615,"5min":13.750999999999999,"15min":26.07},"min":{"1min":8.2759999999999998,"5min":7.2210000000000001,"15min":7.2210000000000001},"max":{"1min":31.048999999999999,"5min":31.048999999999999,"15min":371.17599999999999},"last":19.02},{"interface":"front","average":{"1min":13.039999999999999,"5min":13.173,"15min":19.331},"min":{"1min":7.931,"5min":7.5679999999999996,"15min":7.1020000000000003},"max":{"1min":22.701000000000001,"5min":22.701000000000001,"15min":179.63900000000001},"last":17.422000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509824,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78628,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94293212,"statfs":{"total":96636764160,"available":96556249088,"internally_reserved":0,"allocated":11628544,"data_stored":6358216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":9.6099999999999994,"5min":9.1780000000000008,"15min":20.884},"min":{"1min":5.1840000000000002,"5min":4.7140000000000004,"15min":4.5629999999999997},"max":{"1min":14.326000000000001,"5min":18.608000000000001,"15min":273.28300000000002},"last":5.5659999999999998},{"interface":"front","average":{"1min":8.9429999999999996,"5min":9.4139999999999997,"15min":19.279},"min":{"1min":5.1059999999999999,"5min":4.9880000000000004,"15min":4.9880000000000004},"max":{"1min":16.013999999999999,"5min":18.042000000000002,"15min":278.84500000000003},"last":7.5039999999999996}]},{"osd":2,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":11.167999999999999,"5min":10.718999999999999,"15min":22.236999999999998},"min":{"1min":6.7869999999999999,"5min":5.5469999999999997,"15min":5.5469999999999997},"max":{"1min":17.061,"5min":18.632999999999999,"15min":404.36700000000002},"last":10.840999999999999},{"interface":"front","average":{"1min":15.44,"5min":15.366,"15min":29.544},"min":{"1min":10.481999999999999,"5min":9.4169999999999998,"15min":8.8650000000000002},"max":{"1min":21.687999999999999,"5min":22.27,"15min":407.68000000000001},"last":18.472000000000001}]},{"osd":3,"last update":"Sun Jan 21 05:05:06 2024","interfaces":[{"interface":"back","average":{"1min":26.777000000000001,"5min":17.292000000000002,"15min":25.620999999999999},"min":{"1min":9.8919999999999995,"5min":6.266,"15min":6.266},"max":{"1min":212.34200000000001,"5min":212.34200000000001,"15min":253.25399999999999},"last":19.068000000000001},{"interface":"front","average":{"1min":13.295999999999999,"5min":10.103999999999999,"15min":17.751999999999999},"min":{"1min":5.7930000000000001,"5min":5.7930000000000001,"15min":5.7930000000000001},"max":{"1min":63.148000000000003,"5min":63.148000000000003,"15min":315.92399999999998},"last":11.481}]},{"osd":4,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":10.625999999999999,"5min":11.837,"15min":20.167999999999999},"min":{"1min":7.5839999999999996,"5min":7.0149999999999997,"15min":6.2089999999999996},"max":{"1min":21.039999999999999,"5min":22.332999999999998,"15min":283.37599999999998},"last":9.5380000000000003},{"interface":"front","average":{"1min":16.094999999999999,"5min":16.707000000000001,"15min":28.533000000000001},"min":{"1min":11.058999999999999,"5min":9.7360000000000007,"15min":9.7360000000000007},"max":{"1min":21.434999999999999,"5min":23.916,"15min":344.37200000000001},"last":17.547000000000001}]},{"osd":5,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":13.879,"5min":14.018000000000001,"15min":26.966000000000001},"min":{"1min":7.4290000000000003,"5min":7.4290000000000003,"15min":6.6029999999999998},"max":{"1min":18.428999999999998,"5min":20.844000000000001,"15min":354.04300000000001},"last":20.216000000000001},{"interface":"front","average":{"1min":12.494999999999999,"5min":12.369999999999999,"15min":23.998000000000001},"min":{"1min":8.6600000000000001,"5min":7.7089999999999996,"15min":6.3159999999999998},"max":{"1min":17.739999999999998,"5min":22.294,"15min":296.52499999999998},"last":10.298999999999999}]},{"osd":6,"last update":"Sun Jan 21 05:05:02 2024","interfaces":[{"interface":"back","average":{"1min":25.177,"5min":16.742000000000001,"15min":24.870000000000001},"min":{"1min":10.148,"5min":8.1799999999999997,"15min":6.5659999999999998},"max":{"1min":226.14599999999999,"5min":226.14599999999999,"15min":288.29599999999999},"last":20.986000000000001},{"interface":"front","average":{"1min":27.905999999999999,"5min":19.323,"15min":29.056000000000001},"min":{"1min":11.382,"5min":10.058,"15min":8.8550000000000004},"max":{"1min":236.083,"5min":236.083,"15min":293.56400000000002},"last":21.853000000000002}]},{"osd":7,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":28.417999999999999,"5min":15.923999999999999,"15min":23.308},"min":{"1min":9.2400000000000002,"5min":8.0700000000000003,"15min":7.6509999999999998},"max":{"1min":113.319,"5min":113.319,"15min":329.11200000000002},"last":15.205},{"interface":"front","average":{"1min":33.444000000000003,"5min":19.013000000000002,"15min":28.425999999999998},"min":{"1min":10.021000000000001,"5min":10.021000000000001,"15min":9.3699999999999992},"max":{"1min":115.97,"5min":115.97,"15min":337.24000000000001},"last":16.475000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509827,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82788,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289052,"statfs":{"total":96636764160,"available":96551989248,"internally_reserved":0,"allocated":11628544,"data_stored":6358216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":14.563000000000001,"5min":11.273999999999999,"15min":15.429},"min":{"1min":4.4800000000000004,"5min":4.4800000000000004,"15min":4.343},"max":{"1min":68.804000000000002,"5min":68.804000000000002,"15min":188.334},"last":7.4740000000000002},{"interface":"front","average":{"1min":10.598000000000001,"5min":9.1699999999999999,"15min":14.536},"min":{"1min":5.8540000000000001,"5min":5.0380000000000003,"15min":4.2919999999999998},"max":{"1min":26.100000000000001,"5min":26.100000000000001,"15min":153.12299999999999},"last":9.3000000000000007}]},{"osd":1,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":13.564,"5min":10.316000000000001,"15min":16.317},"min":{"1min":5.6440000000000001,"5min":5.5419999999999998,"15min":5.5419999999999998},"max":{"1min":37.771999999999998,"5min":37.771999999999998,"15min":318.404},"last":11.583},{"interface":"front","average":{"1min":19.129000000000001,"5min":14.199,"15min":21.916},"min":{"1min":10.805999999999999,"5min":7.1699999999999999,"15min":7.1699999999999999},"max":{"1min":58.195,"5min":58.195,"15min":333.91000000000003},"last":15.878}]},{"osd":3,"last update":"Sun Jan 21 05:05:24 2024","interfaces":[{"interface":"back","average":{"1min":17.065000000000001,"5min":12.718,"15min":18.966999999999999},"min":{"1min":7.1639999999999997,"5min":6.9340000000000002,"15min":5.7240000000000002},"max":{"1min":48.807000000000002,"5min":48.807000000000002,"15min":314.58600000000001},"last":12.199999999999999},{"interface":"front","average":{"1min":19.007000000000001,"5min":15.404999999999999,"15min":22.751000000000001},"min":{"1min":9.9469999999999992,"5min":9.7530000000000001,"15min":8.5250000000000004},"max":{"1min":71.736999999999995,"5min":71.736999999999995,"15min":335.71699999999998},"last":10.006}]},{"osd":4,"last update":"Sun Jan 21 05:04:39 2024","interfaces":[{"interface":"back","average":{"1min":12.945,"5min":13.516,"15min":19.908000000000001},"min":{"1min":8.6189999999999998,"5min":8.25,"15min":6.0999999999999996},"max":{"1min":18.849,"5min":23.408000000000001,"15min":313.17200000000003},"last":16.82},{"interface":"front","average":{"1min":15.632999999999999,"5min":15.916,"15min":22.266999999999999},"min":{"1min":10.994,"5min":9.593,"15min":6.1920000000000002},"max":{"1min":18.355,"5min":27.402999999999999,"15min":331.18200000000002},"last":14.096}]},{"osd":5,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":20.053999999999998,"5min":12.343999999999999,"15min":19.91},"min":{"1min":6.8529999999999998,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":166.327,"5min":166.327,"15min":270.91399999999999},"last":8.4440000000000008},{"interface":"front","average":{"1min":24.475999999999999,"5min":17.946000000000002,"15min":24.292999999999999},"min":{"1min":13.491,"5min":10.361000000000001,"15min":8.8279999999999994},"max":{"1min":130.548,"5min":130.548,"15min":259.60399999999998},"last":17.736000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:04:51 2024","interfaces":[{"interface":"back","average":{"1min":17.291,"5min":17.452999999999999,"15min":26.175000000000001},"min":{"1min":12.855,"5min":11.922000000000001,"15min":7.9800000000000004},"max":{"1min":20.937000000000001,"5min":31.047000000000001,"15min":321.79500000000002},"last":19.460000000000001},{"interface":"front","average":{"1min":12.227,"5min":12.388,"15min":21.454999999999998},"min":{"1min":8.5169999999999995,"5min":7.9699999999999998,"15min":7.7229999999999999},"max":{"1min":20.093,"5min":28.638999999999999,"15min":282.54599999999999},"last":12.933}]},{"osd":7,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":24.565000000000001,"5min":19.756,"15min":26.001000000000001},"min":{"1min":16.887,"5min":13.028,"15min":8.8460000000000001},"max":{"1min":67.106999999999999,"5min":67.106999999999999,"15min":312.435},"last":18.59},{"interface":"front","average":{"1min":20.120000000000001,"5min":15.542,"15min":23.835000000000001},"min":{"1min":8.5540000000000003,"5min":8.2370000000000001,"15min":8.2370000000000001},"max":{"1min":97.379000000000005,"5min":97.379000000000005,"15min":256.11900000000003},"last":14.731999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509825,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74864,"kb_used_data":11752,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94296976,"statfs":{"total":96636764160,"available":96560103424,"internally_reserved":0,"allocated":12034048,"data_stored":6756061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:04:45 2024","interfaces":[{"interface":"back","average":{"1min":8.1910000000000007,"5min":8.7469999999999999,"15min":16.152999999999999},"min":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"max":{"1min":17.446000000000002,"5min":27.192,"15min":209.96299999999999},"last":7.2469999999999999},{"interface":"front","average":{"1min":9.2430000000000003,"5min":9.1839999999999993,"15min":17.379000000000001},"min":{"1min":5.3650000000000002,"5min":4.9560000000000004,"15min":4.7549999999999999},"max":{"1min":14.426,"5min":25.323,"15min":244.054},"last":8.2629999999999999}]},{"osd":2,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":25.413,"5min":13.148999999999999,"15min":17.792999999999999},"min":{"1min":8.1020000000000003,"5min":6.202,"15min":4.8819999999999997},"max":{"1min":218.80500000000001,"5min":218.80500000000001,"15min":221.54300000000001},"last":8.0009999999999994},{"interface":"front","average":{"1min":24.704999999999998,"5min":15.188000000000001,"15min":20.256},"min":{"1min":10.317,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":144.12899999999999,"5min":144.12899999999999,"15min":219.34899999999999},"last":13.866}]},{"osd":3,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":14.996,"5min":13.903,"15min":21.224},"min":{"1min":7.3200000000000003,"5min":6.569,"15min":6.569},"max":{"1min":40.685000000000002,"5min":40.685000000000002,"15min":163.58799999999999},"last":17.402000000000001},{"interface":"front","average":{"1min":14.272,"5min":12.506,"15min":19.687999999999999},"min":{"1min":7.9630000000000001,"5min":7.0549999999999997,"15min":6.4809999999999999},"max":{"1min":55.024999999999999,"5min":55.024999999999999,"15min":244.178},"last":15.630000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:04:45 2024","interfaces":[{"interface":"back","average":{"1min":15.436999999999999,"5min":15.552,"15min":21.210000000000001},"min":{"1min":7.6369999999999996,"5min":7.6369999999999996,"15min":7.6369999999999996},"max":{"1min":19.053000000000001,"5min":23.524999999999999,"15min":241.94800000000001},"last":18.265000000000001},{"interface":"front","average":{"1min":10.997,"5min":10.789999999999999,"15min":18.545000000000002},"min":{"1min":7.0789999999999997,"5min":6.7069999999999999,"15min":6.7069999999999999},"max":{"1min":18.260999999999999,"5min":18.260999999999999,"15min":225.60499999999999},"last":12.446999999999999}]},{"osd":5,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":26.631,"5min":16.741,"15min":22.994},"min":{"1min":9.1850000000000005,"5min":9.1850000000000005,"15min":6.8970000000000002},"max":{"1min":156.554,"5min":156.554,"15min":313.36599999999999},"last":10.292999999999999},{"interface":"front","average":{"1min":26.550000000000001,"5min":16.300999999999998,"15min":23.606000000000002},"min":{"1min":8.9369999999999994,"5min":8.9369999999999994,"15min":5.851},"max":{"1min":138.114,"5min":138.114,"15min":298.49599999999998},"last":9.0289999999999999}]},{"osd":6,"last update":"Sun Jan 21 05:04:45 2024","interfaces":[{"interface":"back","average":{"1min":15.303000000000001,"5min":15.319000000000001,"15min":24.856000000000002},"min":{"1min":9.827,"5min":8.8949999999999996,"15min":8.8949999999999996},"max":{"1min":19.283000000000001,"5min":23.902999999999999,"15min":298.51999999999998},"last":11.117000000000001},{"interface":"front","average":{"1min":16.611999999999998,"5min":16.690999999999999,"15min":24.472999999999999},"min":{"1min":11.101000000000001,"5min":10.550000000000001,"15min":7.4880000000000004},"max":{"1min":21.846,"5min":26.234000000000002,"15min":181.01599999999999},"last":13.145}]},{"osd":7,"last update":"Sun Jan 21 05:05:17 2024","interfaces":[{"interface":"back","average":{"1min":21.521000000000001,"5min":16.407,"15min":24.297999999999998},"min":{"1min":11.332000000000001,"5min":9.8070000000000004,"15min":9.3230000000000004},"max":{"1min":107.875,"5min":107.875,"15min":214.24199999999999},"last":11.731},{"interface":"front","average":{"1min":20.009,"5min":17.219999999999999,"15min":24.443000000000001},"min":{"1min":9.5779999999999994,"5min":9.5779999999999994,"15min":9.4120000000000008},"max":{"1min":51.546999999999997,"5min":51.546999999999997,"15min":180.73099999999999},"last":16.562999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575233,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74860,"kb_used_data":11748,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94296980,"statfs":{"total":96636764160,"available":96560107520,"internally_reserved":0,"allocated":12029952,"data_stored":6756056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":23.058,"5min":12.195,"15min":18.460000000000001},"min":{"1min":6.226,"5min":5.2300000000000004,"15min":4.601},"max":{"1min":154.13399999999999,"5min":154.13399999999999,"15min":377.50999999999999},"last":13.148},{"interface":"front","average":{"1min":32.405999999999999,"5min":17.396999999999998,"15min":21.706},"min":{"1min":7.8029999999999999,"5min":5.8559999999999999,"15min":4.4359999999999999},"max":{"1min":193.01400000000001,"5min":193.01400000000001,"15min":224.58600000000001},"last":22.873999999999999}]},{"osd":1,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":24.754999999999999,"5min":14.738,"15min":22.327999999999999},"min":{"1min":6.165,"5min":5.6820000000000004,"15min":4.5380000000000003},"max":{"1min":160.279,"5min":160.279,"15min":368.79199999999997},"last":10.007},{"interface":"front","average":{"1min":28.187000000000001,"5min":13.547000000000001,"15min":19.158999999999999},"min":{"1min":9.9169999999999998,"5min":4.944,"15min":4.3170000000000002},"max":{"1min":156.38499999999999,"5min":156.38499999999999,"15min":225.21000000000001},"last":7.5839999999999996}]},{"osd":2,"last update":"Sun Jan 21 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":28.533000000000001,"5min":14.004,"15min":20.603999999999999},"min":{"1min":7.7069999999999999,"5min":6.6420000000000003,"15min":5.8460000000000001},"max":{"1min":232.78200000000001,"5min":232.78200000000001,"15min":232.78200000000001},"last":11.324999999999999},{"interface":"front","average":{"1min":35.103000000000002,"5min":15.885999999999999,"15min":22.988},"min":{"1min":7.9740000000000002,"5min":6.4199999999999999,"15min":5.8289999999999997},"max":{"1min":288.49099999999999,"5min":288.49099999999999,"15min":309.24599999999998},"last":6.6349999999999998}]},{"osd":3,"last update":"Sun Jan 21 05:04:44 2024","interfaces":[{"interface":"back","average":{"1min":15.166,"5min":15.417,"15min":25.603000000000002},"min":{"1min":11.159000000000001,"5min":8.516,"15min":6.7229999999999999},"max":{"1min":18.838000000000001,"5min":31.256,"15min":375.07799999999997},"last":12.566000000000001},{"interface":"front","average":{"1min":12.089,"5min":11.997999999999999,"15min":22.629999999999999},"min":{"1min":7.6299999999999999,"5min":6.4589999999999996,"15min":6.4589999999999996},"max":{"1min":16.795999999999999,"5min":32.795999999999999,"15min":276.57100000000003},"last":14.372}]},{"osd":4,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":31.202999999999999,"5min":15.868,"15min":24.077000000000002},"min":{"1min":8.9039999999999999,"5min":6.3360000000000003,"15min":6.3360000000000003},"max":{"1min":184.767,"5min":184.767,"15min":272.274},"last":8.3659999999999997},{"interface":"front","average":{"1min":33.640999999999998,"5min":20.071999999999999,"15min":28.475000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":6.8689999999999998},"max":{"1min":204.47499999999999,"5min":204.47499999999999,"15min":378.404},"last":21.052}]},{"osd":6,"last update":"Sun Jan 21 05:04:44 2024","interfaces":[{"interface":"back","average":{"1min":13.416,"5min":12.768000000000001,"15min":23.533999999999999},"min":{"1min":9.0690000000000008,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":19.172000000000001,"5min":32.139000000000003,"15min":296.88499999999999},"last":17.539999999999999},{"interface":"front","average":{"1min":16.545999999999999,"5min":16.481999999999999,"15min":30.184999999999999},"min":{"1min":11.481999999999999,"5min":10.188000000000001,"15min":9.6750000000000007},"max":{"1min":20.914000000000001,"5min":37.082999999999998,"15min":311.10899999999998},"last":18.202999999999999}]},{"osd":7,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":31.890000000000001,"5min":20.614999999999998,"15min":29.431999999999999},"min":{"1min":10.763,"5min":9.9640000000000004,"15min":8.032},"max":{"1min":202.90700000000001,"5min":202.90700000000001,"15min":486.358},"last":21.878},{"interface":"front","average":{"1min":36.167000000000002,"5min":20.385999999999999,"15min":27.478999999999999},"min":{"1min":10.587999999999999,"5min":9.8740000000000006,"15min":9.1039999999999992},"max":{"1min":177.25800000000001,"5min":177.25800000000001,"15min":486.62900000000002},"last":18.789000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T05:05:45.187 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T05:05:45.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.291+0000 7f7ae5fe9640 10 monclient: get_monmap_and_config 2024-01-21T05:05:45.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.291+0000 7f7ae5fe9640 10 monclient: build_initial_monmap 2024-01-21T05:05:45.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.291+0000 7f7ae5fe9640 10 monclient: monmap: 2024-01-21T05:05:45.292 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:45.292 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:45.292 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:45.292660+0000 2024-01-21T05:05:45.292 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:45.292660+0000 2024-01-21T05:05:45.293 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:45.293 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.293 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:45.293 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:45.293 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:45.293 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:45.293 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:45.293 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.292+0000 7f7ae5fe9640 1 Processor -- start 2024-01-21T05:05:45.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.292+0000 7f7ae5fe9640 1 -- start start 2024-01-21T05:05:45.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.292+0000 7f7ae5fe9640 10 monclient: init 2024-01-21T05:05:45.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ae00a8a40 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f7ae00b21c0 0x7f7ae013f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 10 monclient(hunting): picked mon.noname-c con 0x7f7ae00b21c0 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ae00a8e20 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae00aa9b0 con 0x7f7ae00a8e20 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae00aac90 con 0x7f7ae00a8a40 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae00aa7d0 con 0x7f7ae00b21c0 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ae5fe9640 10 monclient(hunting): authenticate will time out at 3303.634375s 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7adffff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7ae00a8e20 0x7f7ae014c7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50168/0 (socket says 172.21.15.69:50168) 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7adffff640 1 -- 172.21.15.69:0/708667282 learned_addr learned my addr 172.21.15.69:0/708667282 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T05:05:45.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.293+0000 7f7ad6ffd640 1 --2- 172.21.15.69:0/708667282 >> v2:172.21.15.144:3300/0 conn(0x7f7ae00b21c0 0x7f7ae013f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:45.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.294+0000 7f7ad6ffd640 10 monclient(hunting): get_auth_request con 0x7f7ae00b21c0 auth_method 0 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.294+0000 7f7ad6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.294+0000 7f7ad6ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.294+0000 7f7ad6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.295+0000 7f7ad6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.295+0000 7f7ad6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.295+0000 7f7ad6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.296+0000 7f7adeffd640 1 -- 172.21.15.69:0/708667282 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453950180 0 0) 0x7f7ae00aa9b0 con 0x7f7ae00a8e20 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.296+0000 7f7adeffd640 10 monclient(hunting): _negotiate 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.296+0000 7f7adeffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.296+0000 7f7adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.296+0000 7f7adeffd640 10 monclient(hunting): my global_id is 17418 2024-01-21T05:05:45.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.296+0000 7f7adeffd640 1 -- 172.21.15.69:0/708667282 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7acc003520 con 0x7f7ae00a8e20 2024-01-21T05:05:45.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.297+0000 7f7ad6ffd640 10 monclient(hunting): handle_auth_done global_id 21529 payload 274 2024-01-21T05:05:45.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.297+0000 7f7ad6ffd640 1 -- 172.21.15.69:0/708667282 >> v1:172.21.15.144:6789/0 conn(0x7f7ae00a8a40 legacy=0x7f7ae0145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.297+0000 7f7ad6ffd640 1 -- 172.21.15.69:0/708667282 >> v1:172.21.15.69:6789/0 conn(0x7f7ae00a8e20 legacy=0x7f7ae014c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.297+0000 7f7ad6ffd640 10 monclient: _finish_hunting 0 2024-01-21T05:05:45.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.297+0000 7f7ad6ffd640 1 monclient: found mon.noname-c 2024-01-21T05:05:45.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.297+0000 7f7ad6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:45.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.297+0000 7f7ad6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T05:05:45.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.297+0000 7f7ad6ffd640 1 -- 172.21.15.69:0/708667282 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae00aaf70 con 0x7f7ae00b21c0 2024-01-21T05:05:45.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.301+0000 7f7ad6ffd640 1 --2- 172.21.15.69:0/708667282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae00b21c0 0x7f7ae013f1a0 secure :-1 s=READY pgs=5215 cs=0 l=1 rev1=1 crypto rx=0x7f7ac80097a0 tx=0x7f7ac802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81fb18451002a3d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:45.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.303+0000 7f7adeffd640 1 -- 172.21.15.69:0/708667282 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac802f810 con 0x7f7ae00b21c0 2024-01-21T05:05:45.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.303+0000 7f7adeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:45.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.303+0000 7f7adeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:05:45.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.303+0000 7f7adeffd640 10 monclient: dump: 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.304+0000 7f7adeffd640 10 monclient: _finish_auth 0 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.304+0000 7f7adeffd640 10 monclient: _check_auth_tickets 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.304+0000 7f7adeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.304+0000 7f7ae5fe9640 5 monclient: authenticate success, global_id 21529 2024-01-21T05:05:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.304+0000 7f7ae5fe9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.305+0000 7f7adeffd640 1 -- 172.21.15.69:0/708667282 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac802f970 con 0x7f7ae00b21c0 2024-01-21T05:05:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.305+0000 7f7adeffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.305+0000 7f7adeffd640 1 -- 172.21.15.69:0/708667282 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac802fce0 con 0x7f7ae00b21c0 2024-01-21T05:05:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.305+0000 7f7adeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.305+0000 7f7adeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.305+0000 7f7adeffd640 10 monclient: dump: 2024-01-21T05:05:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:45.307 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.305+0000 7f7ae5fe9640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.305+0000 7f7ae5fe9640 4 set_mon_vals no callback set 2024-01-21T05:05:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.306+0000 7f7ae5fe9640 10 monclient: shutdown 2024-01-21T05:05:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.306+0000 7f7ae5fe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.306+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/708667282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae00b21c0 msgr2=0x7f7ae013f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.306+0000 7f7ae5fe9640 1 --2- 172.21.15.69:0/708667282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae00b21c0 0x7f7ae013f1a0 secure :-1 s=READY pgs=5215 cs=0 l=1 rev1=1 crypto rx=0x7f7ac80097a0 tx=0x7f7ac802b9f0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.306+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/708667282 shutdown_connections 2024-01-21T05:05:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.306+0000 7f7ae5fe9640 1 --2- 172.21.15.69:0/708667282 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae00b21c0 0x7f7ae013f1a0 unknown :-1 s=CLOSED pgs=5215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.306+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/708667282 >> 172.21.15.69:0/708667282 conn(0x7f7ae0066f50 msgr2=0x7f7ae00b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.306+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/708667282 shutdown_connections 2024-01-21T05:05:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.306+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/708667282 wait complete. 2024-01-21T05:05:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.307+0000 7f7ae5fe9640 10 monclient: build_initial_monmap 2024-01-21T05:05:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.307+0000 7f7ae5fe9640 10 monclient: monmap: 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.307+0000 7f7ae5fe9640 1 librados: starting msgr at 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.307+0000 7f7ae5fe9640 1 librados: starting objecter 2024-01-21T05:05:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.307+0000 7f7ae5fe9640 1 Processor -- start 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.307+0000 7f7ae5fe9640 1 -- start start 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.307+0000 7f7ae5fe9640 1 librados: setting wanted keys 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.307+0000 7f7ae5fe9640 1 librados: calling monclient init 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.307+0000 7f7ae5fe9640 10 monclient: init 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae00a8a40 0x7f7ae0142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 10 monclient(hunting): picked mon.noname-b con 0x7f7ae00a8a40 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ae00a8e20 0x7f7ae01d6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 10 monclient(hunting): picked mon.noname-c con 0x7f7ae00a8e20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ae00b21c0 0x7f7ae00b7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 10 monclient(hunting): picked mon.noname-a con 0x7f7ae00b21c0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae00abc40 con 0x7f7ae00b21c0 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae01d72d0 con 0x7f7ae00a8a40 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae00b7d70 con 0x7f7ae00a8e20 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7ae5fe9640 10 monclient(hunting): authenticate will time out at 3303.649342s 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.308+0000 7f7adffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ae00b21c0 0x7f7ae00b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.309+0000 7f7adffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ae00b21c0 0x7f7ae00b7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41560/0 (socket says 172.21.15.69:41560) 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.309+0000 7f7adffff640 1 -- 172.21.15.69:0/2409088566 learned_addr learned my addr 172.21.15.69:0/2409088566 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.309+0000 7f7adffff640 10 monclient(hunting): get_auth_request con 0x7f7ae00b21c0 auth_method 0 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.309+0000 7f7adffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.309+0000 7f7adffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.309+0000 7f7adffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.309+0000 7f7adffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.309+0000 7f7adffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.309+0000 7f7adffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.309+0000 7f7adf7fe640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae00a8a40 0x7f7ae0142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.310+0000 7f7adf7fe640 10 monclient(hunting): get_auth_request con 0x7f7ae00a8a40 auth_method 0 2024-01-21T05:05:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.310+0000 7f7adf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.310+0000 7f7adf7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.310+0000 7f7adf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.310+0000 7f7adf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.310+0000 7f7adf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.310+0000 7f7adf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:45.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.312+0000 7f7ad6ffd640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ae00a8e20 0x7f7ae01d6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:45.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.312+0000 7f7adffff640 10 monclient(hunting): handle_auth_done global_id 17421 payload 824 2024-01-21T05:05:45.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.312+0000 7f7adffff640 1 -- 172.21.15.69:0/2409088566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ae00a8e20 msgr2=0x7f7ae01d6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.312+0000 7f7adffff640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ae00a8e20 0x7f7ae01d6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.312+0000 7f7adffff640 1 -- 172.21.15.69:0/2409088566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae00a8a40 msgr2=0x7f7ae0142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.312+0000 7f7adffff640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae00a8a40 0x7f7ae0142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.312+0000 7f7adffff640 10 monclient: _finish_hunting 0 2024-01-21T05:05:45.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.312+0000 7f7adffff640 1 monclient: found mon.noname-a 2024-01-21T05:05:45.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.312+0000 7f7adffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:45.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.312+0000 7f7adffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:45.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.312+0000 7f7adffff640 1 -- 172.21.15.69:0/2409088566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac8009400 con 0x7f7ae00b21c0 2024-01-21T05:05:45.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.328+0000 7f7adffff640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ae00b21c0 0x7f7ae00b7830 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800cb70 tx=0x7f7ad800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:45.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.328+0000 7f7adcff9640 10 client.?.objecter ms_handle_connect 0x7f7ae00b21c0 2024-01-21T05:05:45.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.328+0000 7f7adcff9640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:45.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 1 -- 172.21.15.69:0/2409088566 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad800ad40 con 0x7f7ae00b21c0 2024-01-21T05:05:45.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:45.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:45.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 10 monclient: dump: 2024-01-21T05:05:45.333 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:45.333 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 10 monclient: _finish_auth 0 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 10 monclient: _check_auth_tickets 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 1 -- 172.21.15.69:0/2409088566 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad8012400 con 0x7f7ae00b21c0 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7ae5fe9640 5 monclient: authenticate success, global_id 17421 2024-01-21T05:05:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 1 -- 172.21.15.69:0/2409088566 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad800aea0 con 0x7f7ae00b21c0 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7ae5fe9640 10 monclient: _renew_subs 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7ae5fe9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/2409088566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae01d48d0 con 0x7f7ae00b21c0 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7ade7fc640 4 set_mon_vals no callback set 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7adcff9640 10 monclient: dump: 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.335 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:45.336 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:45.336 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:45.336 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7ae5fe9640 10 client.17421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:45.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7ae5fe9640 10 monclient: _renew_subs 2024-01-21T05:05:45.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7ae5fe9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:45.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/2409088566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae00b7050 con 0x7f7ae00b21c0 2024-01-21T05:05:45.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.332+0000 7f7ae5fe9640 1 librados: init done 2024-01-21T05:05:45.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.333+0000 7f7ae5fe9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:45.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.333+0000 7f7ae5fe9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:45.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.333+0000 7f7ae5fe9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:45.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.333+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/2409088566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae004e970 con 0x7f7ae00b21c0 2024-01-21T05:05:45.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.581+0000 7f7adcff9640 1 -- 172.21.15.69:0/2409088566 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f7ad8007070 con 0x7f7ae00b21c0 2024-01-21T05:05:45.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.582+0000 7f7adcff9640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f7ac006c8b0 0x7f7ac006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:45.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.582+0000 7f7adf7fe640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f7ac006c8b0 0x7f7ac006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:45.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.582+0000 7f7adf7fe640 10 monclient: get_auth_request con 0x7f7ac006c8b0 auth_method 0 2024-01-21T05:05:45.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.583+0000 7f7adf7fe640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f7ac006c8b0 0x7f7ac006ed70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0000f60 tx=0x7f7ad0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:45.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.583+0000 7f7adcff9640 10 client.17421.objecter ms_handle_connect 0x7f7ac006c8b0 2024-01-21T05:05:45.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.596+0000 7f7adcff9640 1 -- 172.21.15.69:0/2409088566 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1109..1109 src has 1032..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f7ad80586d0 con 0x7f7ae00b21c0 2024-01-21T05:05:45.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.596+0000 7f7adcff9640 10 client.17421.objecter ms_dispatch 0x7f7ae0000b90 osd_map(1109..1109 src has 1032..1109) v4 2024-01-21T05:05:45.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.596+0000 7f7adcff9640 3 client.17421.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:45.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.596+0000 7f7adcff9640 3 client.17421.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:45.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.596+0000 7f7adcff9640 20 client.17421.objecter dump_active .. 0 homeless 2024-01-21T05:05:45.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.596+0000 7f7adcff9640 1 -- 172.21.15.69:0/2409088566 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f7ad8054a10 con 0x7f7ae00b21c0 2024-01-21T05:05:45.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.596+0000 7f7adcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:45.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.596+0000 7f7adcff9640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:45.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.762+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/2409088566 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7ae0146270 con 0x7f7ac006c8b0 2024-01-21T05:05:45.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.767+0000 7f7adcff9640 1 -- 172.21.15.69:0/2409088566 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151767 (secure 0 0 0) 0x7f7ad00014a0 con 0x7f7ac006c8b0 2024-01-21T05:05:45.769 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T05:05:45.770 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-01-21T05:05:45.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 10 librados: watch_flush enter 2024-01-21T05:05:45.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 10 librados: watch_flush exit 2024-01-21T05:05:45.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 20 client.17421.objecter shutdown clearing up homeless session... 2024-01-21T05:05:45.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 10 client.17421.objecter successfully canceled tick 2024-01-21T05:05:45.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/2409088566 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f7ac006c8b0 msgr2=0x7f7ac006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f7ac006c8b0 0x7f7ac006ed70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0000f60 tx=0x7f7ad0009040 comp rx=0 tx=0).stop 2024-01-21T05:05:45.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 10 monclient: shutdown 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/2409088566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ae00b21c0 msgr2=0x7f7ae00b7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ae00b21c0 0x7f7ae00b7830 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800cb70 tx=0x7f7ad800a430 comp rx=0 tx=0).stop 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/2409088566 shutdown_connections 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f7ac006c8b0 0x7f7ac006ed70 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ae00b21c0 0x7f7ae00b7830 unknown :-1 s=CLOSED pgs=3485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f7ae00a8e20 0x7f7ae01d6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 --2- 172.21.15.69:0/2409088566 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7ae00a8a40 0x7f7ae0142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/2409088566 >> 172.21.15.69:0/2409088566 conn(0x7f7ae0066f50 msgr2=0x7f7ae013cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/2409088566 shutdown_connections 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 -- 172.21.15.69:0/2409088566 wait complete. 2024-01-21T05:05:45.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.770+0000 7f7ae5fe9640 1 librados: shutdown 2024-01-21T05:05:45.782 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":2079,"stamp":"2024-01-21T05:05:44.360791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":609024,"kb_used_data":91648,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754365696,"statfs":{"total":773094113280,"available":772470472704,"internally_reserved":0,"allocated":93847552,"data_stored":51661426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":529714736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":-2,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.328529"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":1046,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.057494+0000","last_change":"2024-01-21T04:34:59.893360+0000","last_active":"2024-01-21T05:05:25.057494+0000","last_peered":"2024-01-21T05:05:25.057494+0000","last_clean":"2024-01-21T05:05:25.057494+0000","last_became_active":"2024-01-21T04:34:59.874210+0000","last_became_peered":"2024-01-21T04:34:59.874210+0000","last_unstale":"2024-01-21T05:05:25.057494+0000","last_undegraded":"2024-01-21T05:05:25.057494+0000","last_fullsized":"2024-01-21T05:05:25.057494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:03:39.790631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1063,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.109554+0000","last_change":"2024-01-21T04:35:19.549398+0000","last_active":"2024-01-21T05:05:25.109554+0000","last_peered":"2024-01-21T05:05:25.109554+0000","last_clean":"2024-01-21T05:05:25.109554+0000","last_became_active":"2024-01-21T04:35:19.542355+0000","last_became_peered":"2024-01-21T04:35:19.542355+0000","last_unstale":"2024-01-21T05:05:25.109554+0000","last_undegraded":"2024-01-21T05:05:25.109554+0000","last_fullsized":"2024-01-21T05:05:25.109554+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:17:21.163426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1085,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.257683+0000","last_change":"2024-01-21T04:35:21.254053+0000","last_active":"2024-01-21T05:05:25.257683+0000","last_peered":"2024-01-21T05:05:25.257683+0000","last_clean":"2024-01-21T05:05:25.257683+0000","last_became_active":"2024-01-21T04:35:21.234113+0000","last_became_peered":"2024-01-21T04:35:21.234113+0000","last_unstale":"2024-01-21T05:05:25.257683+0000","last_undegraded":"2024-01-21T05:05:25.257683+0000","last_fullsized":"2024-01-21T05:05:25.257683+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:52:02.299953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1050,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.013476+0000","last_change":"2024-01-21T04:35:23.469523+0000","last_active":"2024-01-21T05:05:25.013476+0000","last_peered":"2024-01-21T05:05:25.013476+0000","last_clean":"2024-01-21T05:05:25.013476+0000","last_became_active":"2024-01-21T04:35:23.463341+0000","last_became_peered":"2024-01-21T04:35:23.463341+0000","last_unstale":"2024-01-21T05:05:25.013476+0000","last_undegraded":"2024-01-21T05:05:25.013476+0000","last_fullsized":"2024-01-21T05:05:25.013476+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:40:34.277268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1050,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.952116+0000","last_change":"2024-01-21T04:35:16.948777+0000","last_active":"2024-01-21T05:05:24.952116+0000","last_peered":"2024-01-21T05:05:24.952116+0000","last_clean":"2024-01-21T05:05:24.952116+0000","last_became_active":"2024-01-21T04:35:16.942042+0000","last_became_peered":"2024-01-21T04:35:16.942042+0000","last_unstale":"2024-01-21T05:05:24.952116+0000","last_undegraded":"2024-01-21T05:05:24.952116+0000","last_fullsized":"2024-01-21T05:05:24.952116+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:15:32.933069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"116'7","reported_seq":1077,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.918924+0000","last_change":"2024-01-21T04:35:25.379657+0000","last_active":"2024-01-21T05:05:24.918924+0000","last_peered":"2024-01-21T05:05:24.918924+0000","last_clean":"2024-01-21T05:05:24.918924+0000","last_became_active":"2024-01-21T04:35:25.364561+0000","last_became_peered":"2024-01-21T04:35:25.364561+0000","last_unstale":"2024-01-21T05:05:24.918924+0000","last_undegraded":"2024-01-21T05:05:24.918924+0000","last_fullsized":"2024-01-21T05:05:24.918924+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:49:49.587513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1084,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.251316+0000","last_change":"2024-01-21T04:34:53.473895+0000","last_active":"2024-01-21T05:05:25.251316+0000","last_peered":"2024-01-21T05:05:25.251316+0000","last_clean":"2024-01-21T05:05:25.251316+0000","last_became_active":"2024-01-21T04:34:53.465163+0000","last_became_peered":"2024-01-21T04:34:53.465163+0000","last_unstale":"2024-01-21T05:05:25.251316+0000","last_undegraded":"2024-01-21T05:05:25.251316+0000","last_fullsized":"2024-01-21T05:05:25.251316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:37:17.441183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1048,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.938561+0000","last_change":"2024-01-21T04:35:02.604107+0000","last_active":"2024-01-21T05:05:24.938561+0000","last_peered":"2024-01-21T05:05:24.938561+0000","last_clean":"2024-01-21T05:05:24.938561+0000","last_became_active":"2024-01-21T04:35:02.595993+0000","last_became_peered":"2024-01-21T04:35:02.595993+0000","last_unstale":"2024-01-21T05:05:24.938561+0000","last_undegraded":"2024-01-21T05:05:24.938561+0000","last_fullsized":"2024-01-21T05:05:24.938561+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:05:40.627686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1084,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.237542+0000","last_change":"2024-01-21T04:35:21.226772+0000","last_active":"2024-01-21T05:05:25.237542+0000","last_peered":"2024-01-21T05:05:25.237542+0000","last_clean":"2024-01-21T05:05:25.237542+0000","last_became_active":"2024-01-21T04:35:21.218816+0000","last_became_peered":"2024-01-21T04:35:21.218816+0000","last_unstale":"2024-01-21T05:05:25.237542+0000","last_undegraded":"2024-01-21T05:05:25.237542+0000","last_fullsized":"2024-01-21T05:05:25.237542+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:07:02.731410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1048,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.037035+0000","last_change":"2024-01-21T04:34:53.062240+0000","last_active":"2024-01-21T05:05:25.037035+0000","last_peered":"2024-01-21T05:05:25.037035+0000","last_clean":"2024-01-21T05:05:25.037035+0000","last_became_active":"2024-01-21T04:34:53.056955+0000","last_became_peered":"2024-01-21T04:34:53.056955+0000","last_unstale":"2024-01-21T05:05:25.037035+0000","last_undegraded":"2024-01-21T05:05:25.037035+0000","last_fullsized":"2024-01-21T05:05:25.037035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:03:25.878354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1065,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.081092+0000","last_change":"2024-01-21T04:35:00.848090+0000","last_active":"2024-01-21T05:05:25.081092+0000","last_peered":"2024-01-21T05:05:25.081092+0000","last_clean":"2024-01-21T05:05:25.081092+0000","last_became_active":"2024-01-21T04:35:00.838638+0000","last_became_peered":"2024-01-21T04:35:00.838638+0000","last_unstale":"2024-01-21T05:05:25.081092+0000","last_undegraded":"2024-01-21T05:05:25.081092+0000","last_fullsized":"2024-01-21T05:05:25.081092+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:25:11.256572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1046,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.049467+0000","last_change":"2024-01-21T04:35:28.444122+0000","last_active":"2024-01-21T05:05:25.049467+0000","last_peered":"2024-01-21T05:05:25.049467+0000","last_clean":"2024-01-21T05:05:25.049467+0000","last_became_active":"2024-01-21T04:35:28.438809+0000","last_became_peered":"2024-01-21T04:35:28.438809+0000","last_unstale":"2024-01-21T05:05:25.049467+0000","last_undegraded":"2024-01-21T05:05:25.049467+0000","last_fullsized":"2024-01-21T05:05:25.049467+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:27:38.339736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1049,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.984819+0000","last_change":"2024-01-21T04:35:00.835300+0000","last_active":"2024-01-21T05:05:24.984819+0000","last_peered":"2024-01-21T05:05:24.984819+0000","last_clean":"2024-01-21T05:05:24.984819+0000","last_became_active":"2024-01-21T04:35:00.824479+0000","last_became_peered":"2024-01-21T04:35:00.824479+0000","last_unstale":"2024-01-21T05:05:24.984819+0000","last_undegraded":"2024-01-21T05:05:24.984819+0000","last_fullsized":"2024-01-21T05:05:24.984819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:11:26.198997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":1179,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.909621+0000","last_change":"2024-01-21T04:33:33.164364+0000","last_active":"2024-01-21T05:05:24.909621+0000","last_peered":"2024-01-21T05:05:24.909621+0000","last_clean":"2024-01-21T05:05:24.909621+0000","last_became_active":"2024-01-21T04:33:33.133732+0000","last_became_peered":"2024-01-21T04:33:33.133732+0000","last_unstale":"2024-01-21T05:05:24.909621+0000","last_undegraded":"2024-01-21T05:05:24.909621+0000","last_fullsized":"2024-01-21T05:05:24.909621+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:29.982915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:29.982915+0000","last_clean_scrub_stamp":"2024-01-21T04:33:29.982915+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:02:00.259080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1057,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.958852+0000","last_change":"2024-01-21T04:33:48.369523+0000","last_active":"2024-01-21T05:05:24.958852+0000","last_peered":"2024-01-21T05:05:24.958852+0000","last_clean":"2024-01-21T05:05:24.958852+0000","last_became_active":"2024-01-21T04:33:48.364387+0000","last_became_peered":"2024-01-21T04:33:48.364387+0000","last_unstale":"2024-01-21T05:05:24.958852+0000","last_undegraded":"2024-01-21T05:05:24.958852+0000","last_fullsized":"2024-01-21T05:05:24.958852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:06:07.142172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1046,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.060352+0000","last_change":"2024-01-21T04:34:54.460251+0000","last_active":"2024-01-21T05:05:25.060352+0000","last_peered":"2024-01-21T05:05:25.060352+0000","last_clean":"2024-01-21T05:05:25.060352+0000","last_became_active":"2024-01-21T04:34:54.452154+0000","last_became_peered":"2024-01-21T04:34:54.452154+0000","last_unstale":"2024-01-21T05:05:25.060352+0000","last_undegraded":"2024-01-21T05:05:25.060352+0000","last_fullsized":"2024-01-21T05:05:25.060352+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:03:09.023273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1057,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.888382+0000","last_change":"2024-01-21T04:33:47.762316+0000","last_active":"2024-01-21T05:05:24.888382+0000","last_peered":"2024-01-21T05:05:24.888382+0000","last_clean":"2024-01-21T05:05:24.888382+0000","last_became_active":"2024-01-21T04:33:47.734733+0000","last_became_peered":"2024-01-21T04:33:47.734733+0000","last_unstale":"2024-01-21T05:05:24.888382+0000","last_undegraded":"2024-01-21T05:05:24.888382+0000","last_fullsized":"2024-01-21T05:05:24.888382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:16:54.133000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1049,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.016911+0000","last_change":"2024-01-21T04:35:05.297742+0000","last_active":"2024-01-21T05:05:25.016911+0000","last_peered":"2024-01-21T05:05:25.016911+0000","last_clean":"2024-01-21T05:05:25.016911+0000","last_became_active":"2024-01-21T04:35:05.290888+0000","last_became_peered":"2024-01-21T04:35:05.290888+0000","last_unstale":"2024-01-21T05:05:25.016911+0000","last_undegraded":"2024-01-21T05:05:25.016911+0000","last_fullsized":"2024-01-21T05:05:25.016911+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:51:16.216402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1053,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.991808+0000","last_change":"2024-01-21T04:33:48.697481+0000","last_active":"2024-01-21T05:05:24.991808+0000","last_peered":"2024-01-21T05:05:24.991808+0000","last_clean":"2024-01-21T05:05:24.991808+0000","last_became_active":"2024-01-21T04:33:48.670864+0000","last_became_peered":"2024-01-21T04:33:48.670864+0000","last_unstale":"2024-01-21T05:05:24.991808+0000","last_undegraded":"2024-01-21T05:05:24.991808+0000","last_fullsized":"2024-01-21T05:05:24.991808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:00:07.393617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1046,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.013422+0000","last_change":"2024-01-21T04:35:07.348347+0000","last_active":"2024-01-21T05:05:25.013422+0000","last_peered":"2024-01-21T05:05:25.013422+0000","last_clean":"2024-01-21T05:05:25.013422+0000","last_became_active":"2024-01-21T04:35:07.340277+0000","last_became_peered":"2024-01-21T04:35:07.340277+0000","last_unstale":"2024-01-21T05:05:25.013422+0000","last_undegraded":"2024-01-21T05:05:25.013422+0000","last_fullsized":"2024-01-21T05:05:25.013422+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:33:05.933710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1059,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.943953+0000","last_change":"2024-01-21T04:37:13.894971+0000","last_active":"2024-01-21T05:05:24.943953+0000","last_peered":"2024-01-21T05:05:24.943953+0000","last_clean":"2024-01-21T05:05:24.943953+0000","last_became_active":"2024-01-21T04:33:48.015845+0000","last_became_peered":"2024-01-21T04:33:48.015845+0000","last_unstale":"2024-01-21T05:05:24.943953+0000","last_undegraded":"2024-01-21T05:05:24.943953+0000","last_fullsized":"2024-01-21T05:05:24.943953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:42:22.571268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1048,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.907668+0000","last_change":"2024-01-21T04:35:16.132218+0000","last_active":"2024-01-21T05:05:24.907668+0000","last_peered":"2024-01-21T05:05:24.907668+0000","last_clean":"2024-01-21T05:05:24.907668+0000","last_became_active":"2024-01-21T04:35:16.127034+0000","last_became_peered":"2024-01-21T04:35:16.127034+0000","last_unstale":"2024-01-21T05:05:24.907668+0000","last_undegraded":"2024-01-21T05:05:24.907668+0000","last_fullsized":"2024-01-21T05:05:24.907668+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:29:27.366404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1054,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.059062+0000","last_change":"2024-01-21T04:33:48.068776+0000","last_active":"2024-01-21T05:05:25.059062+0000","last_peered":"2024-01-21T05:05:25.059062+0000","last_clean":"2024-01-21T05:05:25.059062+0000","last_became_active":"2024-01-21T04:33:48.032152+0000","last_became_peered":"2024-01-21T04:33:48.032152+0000","last_unstale":"2024-01-21T05:05:25.059062+0000","last_undegraded":"2024-01-21T05:05:25.059062+0000","last_fullsized":"2024-01-21T05:05:25.059062+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:12:43.256934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1046,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.002979+0000","last_change":"2024-01-21T04:34:55.542633+0000","last_active":"2024-01-21T05:05:25.002979+0000","last_peered":"2024-01-21T05:05:25.002979+0000","last_clean":"2024-01-21T05:05:25.002979+0000","last_became_active":"2024-01-21T04:34:55.527971+0000","last_became_peered":"2024-01-21T04:34:55.527971+0000","last_unstale":"2024-01-21T05:05:25.002979+0000","last_undegraded":"2024-01-21T05:05:25.002979+0000","last_fullsized":"2024-01-21T05:05:25.002979+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:19:42.501825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1056,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.986390+0000","last_change":"2024-01-21T04:33:47.840628+0000","last_active":"2024-01-21T05:05:24.986390+0000","last_peered":"2024-01-21T05:05:24.986390+0000","last_clean":"2024-01-21T05:05:24.986390+0000","last_became_active":"2024-01-21T04:33:47.835639+0000","last_became_peered":"2024-01-21T04:33:47.835639+0000","last_unstale":"2024-01-21T05:05:24.986390+0000","last_undegraded":"2024-01-21T05:05:24.986390+0000","last_fullsized":"2024-01-21T05:05:24.986390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:55:33.768752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1047,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.977642+0000","last_change":"2024-01-21T04:35:26.902591+0000","last_active":"2024-01-21T05:05:24.977642+0000","last_peered":"2024-01-21T05:05:24.977642+0000","last_clean":"2024-01-21T05:05:24.977642+0000","last_became_active":"2024-01-21T04:35:26.889439+0000","last_became_peered":"2024-01-21T04:35:26.889439+0000","last_unstale":"2024-01-21T05:05:24.977642+0000","last_undegraded":"2024-01-21T05:05:24.977642+0000","last_fullsized":"2024-01-21T05:05:24.977642+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:36:16.061320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1052,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.954450+0000","last_change":"2024-01-21T04:33:48.112047+0000","last_active":"2024-01-21T05:05:24.954450+0000","last_peered":"2024-01-21T05:05:24.954450+0000","last_clean":"2024-01-21T05:05:24.954450+0000","last_became_active":"2024-01-21T04:33:48.080898+0000","last_became_peered":"2024-01-21T04:33:48.080898+0000","last_unstale":"2024-01-21T05:05:24.954450+0000","last_undegraded":"2024-01-21T05:05:24.954450+0000","last_fullsized":"2024-01-21T05:05:24.954450+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:54:29.756588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1064,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.082836+0000","last_change":"2024-01-21T04:35:36.906825+0000","last_active":"2024-01-21T05:05:25.082836+0000","last_peered":"2024-01-21T05:05:25.082836+0000","last_clean":"2024-01-21T05:05:25.082836+0000","last_became_active":"2024-01-21T04:35:36.901570+0000","last_became_peered":"2024-01-21T04:35:36.901570+0000","last_unstale":"2024-01-21T05:05:25.082836+0000","last_undegraded":"2024-01-21T05:05:25.082836+0000","last_fullsized":"2024-01-21T05:05:25.082836+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:17:42.698152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1055,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.015022+0000","last_change":"2024-01-21T04:33:48.051441+0000","last_active":"2024-01-21T05:05:25.015022+0000","last_peered":"2024-01-21T05:05:25.015022+0000","last_clean":"2024-01-21T05:05:25.015022+0000","last_became_active":"2024-01-21T04:33:48.046404+0000","last_became_peered":"2024-01-21T04:33:48.046404+0000","last_unstale":"2024-01-21T05:05:25.015022+0000","last_undegraded":"2024-01-21T05:05:25.015022+0000","last_fullsized":"2024-01-21T05:05:25.015022+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:19:24.446344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1047,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.998878+0000","last_change":"2024-01-21T04:35:19.621072+0000","last_active":"2024-01-21T05:05:24.998878+0000","last_peered":"2024-01-21T05:05:24.998878+0000","last_clean":"2024-01-21T05:05:24.998878+0000","last_became_active":"2024-01-21T04:35:19.611575+0000","last_became_peered":"2024-01-21T04:35:19.611575+0000","last_unstale":"2024-01-21T05:05:24.998878+0000","last_undegraded":"2024-01-21T05:05:24.998878+0000","last_fullsized":"2024-01-21T05:05:24.998878+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:09:33.751134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1082,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.249530+0000","last_change":"2024-01-21T04:34:53.506620+0000","last_active":"2024-01-21T05:05:25.249530+0000","last_peered":"2024-01-21T05:05:25.249530+0000","last_clean":"2024-01-21T05:05:25.249530+0000","last_became_active":"2024-01-21T04:34:53.495212+0000","last_became_peered":"2024-01-21T04:34:53.495212+0000","last_unstale":"2024-01-21T05:05:25.249530+0000","last_undegraded":"2024-01-21T05:05:25.249530+0000","last_fullsized":"2024-01-21T05:05:25.249530+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:38:49.385646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1045,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.990199+0000","last_change":"2024-01-21T04:34:58.579556+0000","last_active":"2024-01-21T05:05:24.990199+0000","last_peered":"2024-01-21T05:05:24.990199+0000","last_clean":"2024-01-21T05:05:24.990199+0000","last_became_active":"2024-01-21T04:34:58.492357+0000","last_became_peered":"2024-01-21T04:34:58.492357+0000","last_unstale":"2024-01-21T05:05:24.990199+0000","last_undegraded":"2024-01-21T05:05:24.990199+0000","last_fullsized":"2024-01-21T05:05:24.990199+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:55:43.736268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1045,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.979947+0000","last_change":"2024-01-21T04:35:26.240386+0000","last_active":"2024-01-21T05:05:24.979947+0000","last_peered":"2024-01-21T05:05:24.979947+0000","last_clean":"2024-01-21T05:05:24.979947+0000","last_became_active":"2024-01-21T04:35:25.928650+0000","last_became_peered":"2024-01-21T04:35:25.928650+0000","last_unstale":"2024-01-21T05:05:24.979947+0000","last_undegraded":"2024-01-21T05:05:24.979947+0000","last_fullsized":"2024-01-21T05:05:24.979947+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:38:53.872497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1062,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.970780+0000","last_change":"2024-01-21T04:35:00.395084+0000","last_active":"2024-01-21T05:05:24.970780+0000","last_peered":"2024-01-21T05:05:24.970780+0000","last_clean":"2024-01-21T05:05:24.970780+0000","last_became_active":"2024-01-21T04:35:00.387495+0000","last_became_peered":"2024-01-21T04:35:00.387495+0000","last_unstale":"2024-01-21T05:05:24.970780+0000","last_undegraded":"2024-01-21T05:05:24.970780+0000","last_fullsized":"2024-01-21T05:05:24.970780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:17:14.645461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1082,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.255370+0000","last_change":"2024-01-21T04:35:16.819308+0000","last_active":"2024-01-21T05:05:25.255370+0000","last_peered":"2024-01-21T05:05:25.255370+0000","last_clean":"2024-01-21T05:05:25.255370+0000","last_became_active":"2024-01-21T04:35:16.679256+0000","last_became_peered":"2024-01-21T04:35:16.679256+0000","last_unstale":"2024-01-21T05:05:25.255370+0000","last_undegraded":"2024-01-21T05:05:25.255370+0000","last_fullsized":"2024-01-21T05:05:25.255370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:48:12.462617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1062,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.959859+0000","last_change":"2024-01-21T04:35:25.009419+0000","last_active":"2024-01-21T05:05:24.959859+0000","last_peered":"2024-01-21T05:05:24.959859+0000","last_clean":"2024-01-21T05:05:24.959859+0000","last_became_active":"2024-01-21T04:35:25.004049+0000","last_became_peered":"2024-01-21T04:35:25.004049+0000","last_unstale":"2024-01-21T05:05:24.959859+0000","last_undegraded":"2024-01-21T05:05:24.959859+0000","last_fullsized":"2024-01-21T05:05:24.959859+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:44:01.746389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1048,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.015015+0000","last_change":"2024-01-21T04:35:23.577107+0000","last_active":"2024-01-21T05:05:25.015015+0000","last_peered":"2024-01-21T05:05:25.015015+0000","last_clean":"2024-01-21T05:05:25.015015+0000","last_became_active":"2024-01-21T04:35:23.567833+0000","last_became_peered":"2024-01-21T04:35:23.567833+0000","last_unstale":"2024-01-21T05:05:25.015015+0000","last_undegraded":"2024-01-21T05:05:25.015015+0000","last_fullsized":"2024-01-21T05:05:25.015015+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:50:25.014755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1047,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.021801+0000","last_change":"2024-01-21T04:34:57.160123+0000","last_active":"2024-01-21T05:05:25.021801+0000","last_peered":"2024-01-21T05:05:25.021801+0000","last_clean":"2024-01-21T05:05:25.021801+0000","last_became_active":"2024-01-21T04:34:57.153977+0000","last_became_peered":"2024-01-21T04:34:57.153977+0000","last_unstale":"2024-01-21T05:05:25.021801+0000","last_undegraded":"2024-01-21T05:05:25.021801+0000","last_fullsized":"2024-01-21T05:05:25.021801+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:18:26.237354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1047,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.906165+0000","last_change":"2024-01-21T04:35:28.601417+0000","last_active":"2024-01-21T05:05:24.906165+0000","last_peered":"2024-01-21T05:05:24.906165+0000","last_clean":"2024-01-21T05:05:24.906165+0000","last_became_active":"2024-01-21T04:35:28.583632+0000","last_became_peered":"2024-01-21T04:35:28.583632+0000","last_unstale":"2024-01-21T05:05:24.906165+0000","last_undegraded":"2024-01-21T05:05:24.906165+0000","last_fullsized":"2024-01-21T05:05:24.906165+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:35:42.169701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1082,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.225971+0000","last_change":"2024-01-21T04:35:16.742449+0000","last_active":"2024-01-21T05:05:25.225971+0000","last_peered":"2024-01-21T05:05:25.225971+0000","last_clean":"2024-01-21T05:05:25.225971+0000","last_became_active":"2024-01-21T04:35:16.734793+0000","last_became_peered":"2024-01-21T04:35:16.734793+0000","last_unstale":"2024-01-21T05:05:25.225971+0000","last_undegraded":"2024-01-21T05:05:25.225971+0000","last_fullsized":"2024-01-21T05:05:25.225971+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:19:56.812663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1064,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.059570+0000","last_change":"2024-01-21T04:35:14.711183+0000","last_active":"2024-01-21T05:05:25.059570+0000","last_peered":"2024-01-21T05:05:25.059570+0000","last_clean":"2024-01-21T05:05:25.059570+0000","last_became_active":"2024-01-21T04:35:14.685816+0000","last_became_peered":"2024-01-21T04:35:14.685816+0000","last_unstale":"2024-01-21T05:05:25.059570+0000","last_undegraded":"2024-01-21T05:05:25.059570+0000","last_fullsized":"2024-01-21T05:05:25.059570+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:04:28.368206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":21,"seq":90194313604,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74468,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297372,"statfs":{"total":96636764160,"available":96560508928,"internally_reserved":0,"allocated":11628544,"data_stored":6358216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:02 2024","interfaces":[{"interface":"back","average":{"1min":16.902000000000001,"5min":11.343,"15min":17.483000000000001},"min":{"1min":6.3620000000000001,"5min":5.4320000000000004,"15min":4.5259999999999998},"max":{"1min":130.69499999999999,"5min":130.69499999999999,"15min":305.86900000000003},"last":17.004999999999999},{"interface":"front","average":{"1min":19.975999999999999,"5min":14.539,"15min":20.905000000000001},"min":{"1min":10.542999999999999,"5min":8.0370000000000008,"15min":4.6710000000000003},"max":{"1min":121.011,"5min":121.011,"15min":281.79000000000002},"last":19.651}]},{"osd":1,"last update":"Sun Jan 21 05:05:28 2024","interfaces":[{"interface":"back","average":{"1min":30.696999999999999,"5min":16.850999999999999,"15min":19.510000000000002},"min":{"1min":9.0579999999999998,"5min":7.6550000000000002,"15min":6.1280000000000001},"max":{"1min":151.977,"5min":151.977,"15min":249.82599999999999},"last":20.242999999999999},{"interface":"front","average":{"1min":21.411999999999999,"5min":10.917,"15min":14.314},"min":{"1min":5.5709999999999997,"5min":4.8860000000000001,"15min":4.4429999999999996},"max":{"1min":133.238,"5min":133.238,"15min":185.57599999999999},"last":12.459}]},{"osd":2,"last update":"Sun Jan 21 05:05:16 2024","interfaces":[{"interface":"back","average":{"1min":18.077000000000002,"5min":11.068,"15min":16.349},"min":{"1min":6.9770000000000003,"5min":6.4000000000000004,"15min":6.4000000000000004},"max":{"1min":102.258,"5min":102.258,"15min":260.14999999999998},"last":13.958},{"interface":"front","average":{"1min":24.271000000000001,"5min":15.379,"15min":22.163},"min":{"1min":9.6739999999999995,"5min":7.8390000000000004,"15min":7.8390000000000004},"max":{"1min":108.218,"5min":108.218,"15min":261.67200000000003},"last":15.444000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:05:07 2024","interfaces":[{"interface":"back","average":{"1min":21.986999999999998,"5min":12.903,"15min":16.385999999999999},"min":{"1min":6.5780000000000003,"5min":6.1299999999999999,"15min":6.0830000000000002},"max":{"1min":173.92099999999999,"5min":173.92099999999999,"15min":179.381},"last":14.923},{"interface":"front","average":{"1min":22.701000000000001,"5min":15.462,"15min":18.68},"min":{"1min":11.153,"5min":7.0839999999999996,"15min":6.766},"max":{"1min":120.02800000000001,"5min":120.02800000000001,"15min":209.67099999999999},"last":17.731999999999999}]},{"osd":5,"last update":"Sun Jan 21 05:04:58 2024","interfaces":[{"interface":"back","average":{"1min":15.048,"5min":15.337,"15min":22.681999999999999},"min":{"1min":11.866,"5min":7.8719999999999999,"15min":6.9800000000000004},"max":{"1min":17.161999999999999,"5min":30.702000000000002,"15min":318.983},"last":22.001000000000001},{"interface":"front","average":{"1min":12.109,"5min":12.140000000000001,"15min":21.193999999999999},"min":{"1min":8.077,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.620999999999999,"5min":21.742000000000001,"15min":341.21699999999998},"last":18.527000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:05:03 2024","interfaces":[{"interface":"back","average":{"1min":23.292000000000002,"5min":17.651,"15min":26.388999999999999},"min":{"1min":13.853999999999999,"5min":10.154999999999999,"15min":7.1859999999999999},"max":{"1min":110.64100000000001,"5min":110.64100000000001,"15min":329.39100000000002},"last":22.591000000000001},{"interface":"front","average":{"1min":24.414000000000001,"5min":19.693999999999999,"15min":27.457000000000001},"min":{"1min":13.991,"5min":11.544,"15min":11.026},"max":{"1min":102.995,"5min":102.995,"15min":330.54899999999998},"last":24.09}]},{"osd":7,"last update":"Sun Jan 21 05:05:28 2024","interfaces":[{"interface":"back","average":{"1min":40.505000000000003,"5min":19.132000000000001,"15min":21.704000000000001},"min":{"1min":9.1530000000000005,"5min":7.8639999999999999,"15min":7.7240000000000002},"max":{"1min":233.38499999999999,"5min":233.38499999999999,"15min":233.38499999999999},"last":20.902999999999999},{"interface":"front","average":{"1min":41.484999999999999,"5min":21.462,"15min":23.84},"min":{"1min":9.9809999999999999,"5min":9.9120000000000008,"15min":9.9120000000000008},"max":{"1min":264.34800000000001,"5min":264.34800000000001,"15min":264.34800000000001},"last":23.416}]}]},{"osd":6,"up_from":20,"seq":85899346308,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74476,"kb_used_data":11364,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297364,"statfs":{"total":96636764160,"available":96560500736,"internally_reserved":0,"allocated":11636736,"data_stored":6358225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":7.9119999999999999,"5min":8.2769999999999992,"15min":18.212},"min":{"1min":4.3540000000000001,"5min":4.109,"15min":4.0830000000000002},"max":{"1min":13.827,"5min":18.007999999999999,"15min":293.416},"last":5.2009999999999996},{"interface":"front","average":{"1min":8.2260000000000009,"5min":8.4949999999999992,"15min":17.937999999999999},"min":{"1min":4.3849999999999998,"5min":4.3849999999999998,"15min":4.2220000000000004},"max":{"1min":11.989000000000001,"5min":19.242000000000001,"15min":258.678},"last":6.7649999999999997}]},{"osd":1,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":12.247,"5min":11.308999999999999,"15min":17.893999999999998},"min":{"1min":7.6929999999999996,"5min":6.2839999999999998,"15min":6.0650000000000004},"max":{"1min":18.242999999999999,"5min":19.047999999999998,"15min":267.21899999999999},"last":12.834},{"interface":"front","average":{"1min":15.005000000000001,"5min":14.324,"15min":23.260000000000002},"min":{"1min":10.436999999999999,"5min":8.6660000000000004,"15min":7.3339999999999996},"max":{"1min":18.088999999999999,"5min":20.760000000000002,"15min":271.875},"last":13.657}]},{"osd":2,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":11.577,"5min":11.223000000000001,"15min":19.917999999999999},"min":{"1min":6.0039999999999996,"5min":6.0039999999999996,"15min":6.0039999999999996},"max":{"1min":19.081,"5min":19.081,"15min":274.54000000000002},"last":16.366},{"interface":"front","average":{"1min":14.459,"5min":13.849,"15min":17.952000000000002},"min":{"1min":7.5869999999999997,"5min":7.5869999999999997,"15min":5.9420000000000002},"max":{"1min":20.527000000000001,"5min":20.527000000000001,"15min":163.05199999999999},"last":18.417000000000002}]},{"osd":3,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":12.615,"5min":12.771000000000001,"15min":18.033999999999999},"min":{"1min":6.7919999999999998,"5min":6.7919999999999998,"15min":6.7549999999999999},"max":{"1min":18.863,"5min":20.734999999999999,"15min":156.47},"last":8.1890000000000001},{"interface":"front","average":{"1min":15.964,"5min":15.805,"15min":22.997},"min":{"1min":10.096,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":19.196999999999999,"5min":20.001999999999999,"15min":275.81},"last":15.608000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":17.013000000000002,"5min":16.920999999999999,"15min":25.783000000000001},"min":{"1min":13.029999999999999,"5min":11.298,"15min":9.625},"max":{"1min":20.202999999999999,"5min":21.173999999999999,"15min":280.101},"last":17.108000000000001},{"interface":"front","average":{"1min":18.748000000000001,"5min":18.567,"15min":22.736999999999998},"min":{"1min":15.997999999999999,"5min":11.999000000000001,"15min":6.4119999999999999},"max":{"1min":21.719000000000001,"5min":22.102,"15min":273.45600000000002},"last":17.837}]},{"osd":5,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":11.468,"15min":21.411999999999999},"min":{"1min":6.5499999999999998,"5min":6.1699999999999999,"15min":6.1699999999999999},"max":{"1min":20.835000000000001,"5min":20.835000000000001,"15min":275.233},"last":9.5609999999999999},{"interface":"front","average":{"1min":14.619,"5min":14.329000000000001,"15min":22.456},"min":{"1min":7.9710000000000001,"5min":7.9710000000000001,"15min":6.9500000000000002},"max":{"1min":23.047999999999998,"5min":23.047999999999998,"15min":288.76499999999999},"last":10.563000000000001}]},{"osd":7,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":12.044,"5min":12.678000000000001,"15min":21.946000000000002},"min":{"1min":8.0860000000000003,"5min":7.5220000000000002,"15min":7.5220000000000002},"max":{"1min":19.792000000000002,"5min":20.399000000000001,"15min":366.68000000000001},"last":11.395},{"interface":"front","average":{"1min":15.031000000000001,"5min":15.369,"15min":22.943999999999999},"min":{"1min":10.625,"5min":9.6790000000000003,"15min":9.6790000000000003},"max":{"1min":20.709,"5min":21.266999999999999,"15min":281.78500000000003},"last":15.943}]}]},{"osd":4,"up_from":20,"seq":85899346308,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74472,"kb_used_data":11360,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297368,"statfs":{"total":96636764160,"available":96560504832,"internally_reserved":0,"allocated":11632640,"data_stored":6358220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":17.173999999999999,"5min":11.048999999999999,"15min":16.907},"min":{"1min":5.7629999999999999,"5min":5.7629999999999999,"15min":4.2409999999999997},"max":{"1min":154.04499999999999,"5min":154.04499999999999,"15min":294.92099999999999},"last":7.7859999999999996},{"interface":"front","average":{"1min":18.936,"5min":14.742000000000001,"15min":19.247},"min":{"1min":8.673,"5min":6.4379999999999997,"15min":4.6699999999999999},"max":{"1min":99.890000000000001,"5min":99.890000000000001,"15min":292.43700000000001},"last":27.164000000000001}]},{"osd":1,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":10.364000000000001,"5min":9.9369999999999994,"15min":15.395},"min":{"1min":6.0350000000000001,"5min":5.1669999999999998,"15min":4.1470000000000002},"max":{"1min":60.579999999999998,"5min":60.579999999999998,"15min":221.84999999999999},"last":21.420999999999999},{"interface":"front","average":{"1min":12.541,"5min":13.403,"15min":17.917000000000002},"min":{"1min":7.2190000000000003,"5min":7.032,"15min":6.3959999999999999},"max":{"1min":22.050000000000001,"5min":36.454999999999998,"15min":299.11599999999999},"last":28.007000000000001}]},{"osd":2,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":13.195,"5min":13.805999999999999,"15min":19.687999999999999},"min":{"1min":7.6289999999999996,"5min":7.0209999999999999,"15min":5.7439999999999998},"max":{"1min":23.762,"5min":30.693999999999999,"15min":293.60199999999998},"last":19.698},{"interface":"front","average":{"1min":13.038,"5min":11.211,"15min":20.07},"min":{"1min":7.532,"5min":6.4829999999999997,"15min":6.4829999999999997},"max":{"1min":20.181999999999999,"5min":34.270000000000003,"15min":303.94799999999998},"last":20.323}]},{"osd":3,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":29.370000000000001,"5min":16.681000000000001,"15min":22.192},"min":{"1min":6.0940000000000003,"5min":6.0940000000000003,"15min":6.0940000000000003},"max":{"1min":226.172,"5min":226.172,"15min":340.21100000000001},"last":25.408999999999999},{"interface":"front","average":{"1min":36.143999999999998,"5min":16.957999999999998,"15min":22.98},"min":{"1min":9.0310000000000006,"5min":6.6529999999999996,"15min":6.5579999999999998},"max":{"1min":211.38300000000001,"5min":211.38300000000001,"15min":302.72500000000002},"last":17.052}]},{"osd":5,"last update":"Sun Jan 21 05:04:58 2024","interfaces":[{"interface":"back","average":{"1min":23.199999999999999,"5min":15.675000000000001,"15min":23.965},"min":{"1min":8.4740000000000002,"5min":7.2510000000000003,"15min":7.2510000000000003},"max":{"1min":206.16800000000001,"5min":206.16800000000001,"15min":305.17200000000003},"last":24.681000000000001},{"interface":"front","average":{"1min":24.972999999999999,"5min":18.754999999999999,"15min":29.27},"min":{"1min":9.9359999999999999,"5min":9.9359999999999999,"15min":9.9359999999999999},"max":{"1min":192.066,"5min":192.066,"15min":326.19900000000001},"last":28.780000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":24.899999999999999,"5min":15.957000000000001,"15min":23.010999999999999},"min":{"1min":6.8769999999999998,"5min":6.7409999999999997,"15min":6.7409999999999997},"max":{"1min":204.32900000000001,"5min":204.32900000000001,"15min":248.09800000000001},"last":18.550999999999998},{"interface":"front","average":{"1min":27.047000000000001,"5min":18.641999999999999,"15min":26.821000000000002},"min":{"1min":7.7800000000000002,"5min":7.7800000000000002,"15min":7.7800000000000002},"max":{"1min":205.94800000000001,"5min":205.94800000000001,"15min":333.46899999999999},"last":22.148}]},{"osd":7,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":28.289999999999999,"5min":17.347000000000001,"15min":20.901},"min":{"1min":9.5459999999999994,"5min":7.3970000000000002,"15min":6.5430000000000001},"max":{"1min":260.63499999999999,"5min":260.63499999999999,"15min":312.41199999999998},"last":23.809000000000001},{"interface":"front","average":{"1min":29.556000000000001,"5min":19.172999999999998,"15min":25.035},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":236.80799999999999,"5min":236.80799999999999,"15min":247.44499999999999},"last":22.821999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74468,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297372,"statfs":{"total":96636764160,"available":96560508928,"internally_reserved":0,"allocated":11628544,"data_stored":6358216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:21 2024","interfaces":[{"interface":"back","average":{"1min":26.605,"5min":13.619999999999999,"15min":20.289000000000001},"min":{"1min":4.9279999999999999,"5min":4.46,"15min":3.847},"max":{"1min":244.893,"5min":244.893,"15min":272.279},"last":10.243},{"interface":"front","average":{"1min":27.978000000000002,"5min":12.045999999999999,"15min":19.513000000000002},"min":{"1min":4.516,"5min":4.0579999999999998,"15min":4.0579999999999998},"max":{"1min":224.22800000000001,"5min":224.22800000000001,"15min":356.99799999999999},"last":4.5419999999999998}]},{"osd":1,"last update":"Sun Jan 21 05:04:54 2024","interfaces":[{"interface":"back","average":{"1min":8.9190000000000005,"5min":8.7639999999999993,"15min":15.289999999999999},"min":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7759999999999998},"max":{"1min":19.866,"5min":19.866,"15min":276.94099999999997},"last":7.2990000000000004},{"interface":"front","average":{"1min":13.728,"5min":13.041,"15min":20.687999999999999},"min":{"1min":7.8600000000000003,"5min":7.4729999999999999,"15min":6.383},"max":{"1min":23.332000000000001,"5min":23.332000000000001,"15min":232.74299999999999},"last":10.858000000000001}]},{"osd":2,"last update":"Sun Jan 21 05:05:21 2024","interfaces":[{"interface":"back","average":{"1min":32.673999999999999,"5min":17.591000000000001,"15min":25.193000000000001},"min":{"1min":8.125,"5min":7.5919999999999996,"15min":7.5919999999999996},"max":{"1min":259.29899999999998,"5min":259.29899999999998,"15min":284.34100000000001},"last":11.44},{"interface":"front","average":{"1min":35.651000000000003,"5min":20.353000000000002,"15min":24.027999999999999},"min":{"1min":14.430999999999999,"5min":11.855,"15min":6.0229999999999997},"max":{"1min":260.87799999999999,"5min":260.87799999999999,"15min":260.87799999999999},"last":12.641}]},{"osd":3,"last update":"Sun Jan 21 05:05:37 2024","interfaces":[{"interface":"back","average":{"1min":41.347000000000001,"5min":21.199000000000002,"15min":26.056000000000001},"min":{"1min":8.8219999999999992,"5min":8.7919999999999998,"15min":7.8109999999999999},"max":{"1min":263.053,"5min":263.053,"15min":276.61399999999998},"last":12.050000000000001},{"interface":"front","average":{"1min":44.539999999999999,"5min":23.657,"15min":29.605},"min":{"1min":14.435,"5min":13.215,"15min":10.497},"max":{"1min":265.154,"5min":265.154,"15min":277.28100000000001},"last":15.707000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:05:37 2024","interfaces":[{"interface":"back","average":{"1min":50.813000000000002,"5min":18.835000000000001,"15min":17.899000000000001},"min":{"1min":7.5730000000000004,"5min":5.8620000000000001,"15min":5.8620000000000001},"max":{"1min":316.40699999999998,"5min":316.40699999999998,"15min":316.40699999999998},"last":8.3409999999999993},{"interface":"front","average":{"1min":45.965000000000003,"5min":23.863,"15min":25.629000000000001},"min":{"1min":15.303000000000001,"5min":12.269,"15min":9.1609999999999996},"max":{"1min":256.67099999999999,"5min":256.67099999999999,"15min":263.23099999999999},"last":16.457000000000001}]},{"osd":5,"last update":"Sun Jan 21 05:05:22 2024","interfaces":[{"interface":"back","average":{"1min":38.140999999999998,"5min":16.295000000000002,"15min":21.081},"min":{"1min":8.4489999999999998,"5min":6.6459999999999999,"15min":6.6429999999999998},"max":{"1min":223.56999999999999,"5min":223.56999999999999,"15min":223.56999999999999},"last":13.917999999999999},{"interface":"front","average":{"1min":40.212000000000003,"5min":19.149000000000001,"15min":23.727},"min":{"1min":8.6560000000000006,"5min":8.6240000000000006,"15min":8.5429999999999993},"max":{"1min":224.81100000000001,"5min":224.81100000000001,"15min":224.81100000000001},"last":14.672000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:04:54 2024","interfaces":[{"interface":"back","average":{"1min":13.615,"5min":13.750999999999999,"15min":26.07},"min":{"1min":8.2759999999999998,"5min":7.2210000000000001,"15min":7.2210000000000001},"max":{"1min":31.048999999999999,"5min":31.048999999999999,"15min":371.17599999999999},"last":19.02},{"interface":"front","average":{"1min":13.039999999999999,"5min":13.173,"15min":19.331},"min":{"1min":7.931,"5min":7.5679999999999996,"15min":7.1020000000000003},"max":{"1min":22.701000000000001,"5min":22.701000000000001,"15min":179.63900000000001},"last":17.422000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509824,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78628,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94293212,"statfs":{"total":96636764160,"available":96556249088,"internally_reserved":0,"allocated":11628544,"data_stored":6358216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":9.6099999999999994,"5min":9.1780000000000008,"15min":20.884},"min":{"1min":5.1840000000000002,"5min":4.7140000000000004,"15min":4.5629999999999997},"max":{"1min":14.326000000000001,"5min":18.608000000000001,"15min":273.28300000000002},"last":5.5659999999999998},{"interface":"front","average":{"1min":8.9429999999999996,"5min":9.4139999999999997,"15min":19.279},"min":{"1min":5.1059999999999999,"5min":4.9880000000000004,"15min":4.9880000000000004},"max":{"1min":16.013999999999999,"5min":18.042000000000002,"15min":278.84500000000003},"last":7.5039999999999996}]},{"osd":2,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":11.167999999999999,"5min":10.718999999999999,"15min":22.236999999999998},"min":{"1min":6.7869999999999999,"5min":5.5469999999999997,"15min":5.5469999999999997},"max":{"1min":17.061,"5min":18.632999999999999,"15min":404.36700000000002},"last":10.840999999999999},{"interface":"front","average":{"1min":15.44,"5min":15.366,"15min":29.544},"min":{"1min":10.481999999999999,"5min":9.4169999999999998,"15min":8.8650000000000002},"max":{"1min":21.687999999999999,"5min":22.27,"15min":407.68000000000001},"last":18.472000000000001}]},{"osd":3,"last update":"Sun Jan 21 05:05:06 2024","interfaces":[{"interface":"back","average":{"1min":26.777000000000001,"5min":17.292000000000002,"15min":25.620999999999999},"min":{"1min":9.8919999999999995,"5min":6.266,"15min":6.266},"max":{"1min":212.34200000000001,"5min":212.34200000000001,"15min":253.25399999999999},"last":19.068000000000001},{"interface":"front","average":{"1min":13.295999999999999,"5min":10.103999999999999,"15min":17.751999999999999},"min":{"1min":5.7930000000000001,"5min":5.7930000000000001,"15min":5.7930000000000001},"max":{"1min":63.148000000000003,"5min":63.148000000000003,"15min":315.92399999999998},"last":11.481}]},{"osd":4,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":10.625999999999999,"5min":11.837,"15min":20.167999999999999},"min":{"1min":7.5839999999999996,"5min":7.0149999999999997,"15min":6.2089999999999996},"max":{"1min":21.039999999999999,"5min":22.332999999999998,"15min":283.37599999999998},"last":9.5380000000000003},{"interface":"front","average":{"1min":16.094999999999999,"5min":16.707000000000001,"15min":28.533000000000001},"min":{"1min":11.058999999999999,"5min":9.7360000000000007,"15min":9.7360000000000007},"max":{"1min":21.434999999999999,"5min":23.916,"15min":344.37200000000001},"last":17.547000000000001}]},{"osd":5,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":13.879,"5min":14.018000000000001,"15min":26.966000000000001},"min":{"1min":7.4290000000000003,"5min":7.4290000000000003,"15min":6.6029999999999998},"max":{"1min":18.428999999999998,"5min":20.844000000000001,"15min":354.04300000000001},"last":20.216000000000001},{"interface":"front","average":{"1min":12.494999999999999,"5min":12.369999999999999,"15min":23.998000000000001},"min":{"1min":8.6600000000000001,"5min":7.7089999999999996,"15min":6.3159999999999998},"max":{"1min":17.739999999999998,"5min":22.294,"15min":296.52499999999998},"last":10.298999999999999}]},{"osd":6,"last update":"Sun Jan 21 05:05:02 2024","interfaces":[{"interface":"back","average":{"1min":25.177,"5min":16.742000000000001,"15min":24.870000000000001},"min":{"1min":10.148,"5min":8.1799999999999997,"15min":6.5659999999999998},"max":{"1min":226.14599999999999,"5min":226.14599999999999,"15min":288.29599999999999},"last":20.986000000000001},{"interface":"front","average":{"1min":27.905999999999999,"5min":19.323,"15min":29.056000000000001},"min":{"1min":11.382,"5min":10.058,"15min":8.8550000000000004},"max":{"1min":236.083,"5min":236.083,"15min":293.56400000000002},"last":21.853000000000002}]},{"osd":7,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":28.417999999999999,"5min":15.923999999999999,"15min":23.308},"min":{"1min":9.2400000000000002,"5min":8.0700000000000003,"15min":7.6509999999999998},"max":{"1min":113.319,"5min":113.319,"15min":329.11200000000002},"last":15.205},{"interface":"front","average":{"1min":33.444000000000003,"5min":19.013000000000002,"15min":28.425999999999998},"min":{"1min":10.021000000000001,"5min":10.021000000000001,"15min":9.3699999999999992},"max":{"1min":115.97,"5min":115.97,"15min":337.24000000000001},"last":16.475000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509827,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82788,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289052,"statfs":{"total":96636764160,"available":96551989248,"internally_reserved":0,"allocated":11628544,"data_stored":6358216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":14.563000000000001,"5min":11.273999999999999,"15min":15.429},"min":{"1min":4.4800000000000004,"5min":4.4800000000000004,"15min":4.343},"max":{"1min":68.804000000000002,"5min":68.804000000000002,"15min":188.334},"last":7.4740000000000002},{"interface":"front","average":{"1min":10.598000000000001,"5min":9.1699999999999999,"15min":14.536},"min":{"1min":5.8540000000000001,"5min":5.0380000000000003,"15min":4.2919999999999998},"max":{"1min":26.100000000000001,"5min":26.100000000000001,"15min":153.12299999999999},"last":9.3000000000000007}]},{"osd":1,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":13.564,"5min":10.316000000000001,"15min":16.317},"min":{"1min":5.6440000000000001,"5min":5.5419999999999998,"15min":5.5419999999999998},"max":{"1min":37.771999999999998,"5min":37.771999999999998,"15min":318.404},"last":11.583},{"interface":"front","average":{"1min":19.129000000000001,"5min":14.199,"15min":21.916},"min":{"1min":10.805999999999999,"5min":7.1699999999999999,"15min":7.1699999999999999},"max":{"1min":58.195,"5min":58.195,"15min":333.91000000000003},"last":15.878}]},{"osd":3,"last update":"Sun Jan 21 05:05:24 2024","interfaces":[{"interface":"back","average":{"1min":17.065000000000001,"5min":12.718,"15min":18.966999999999999},"min":{"1min":7.1639999999999997,"5min":6.9340000000000002,"15min":5.7240000000000002},"max":{"1min":48.807000000000002,"5min":48.807000000000002,"15min":314.58600000000001},"last":12.199999999999999},{"interface":"front","average":{"1min":19.007000000000001,"5min":15.404999999999999,"15min":22.751000000000001},"min":{"1min":9.9469999999999992,"5min":9.7530000000000001,"15min":8.5250000000000004},"max":{"1min":71.736999999999995,"5min":71.736999999999995,"15min":335.71699999999998},"last":10.006}]},{"osd":4,"last update":"Sun Jan 21 05:04:39 2024","interfaces":[{"interface":"back","average":{"1min":12.945,"5min":13.516,"15min":19.908000000000001},"min":{"1min":8.6189999999999998,"5min":8.25,"15min":6.0999999999999996},"max":{"1min":18.849,"5min":23.408000000000001,"15min":313.17200000000003},"last":16.82},{"interface":"front","average":{"1min":15.632999999999999,"5min":15.916,"15min":22.266999999999999},"min":{"1min":10.994,"5min":9.593,"15min":6.1920000000000002},"max":{"1min":18.355,"5min":27.402999999999999,"15min":331.18200000000002},"last":14.096}]},{"osd":5,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":20.053999999999998,"5min":12.343999999999999,"15min":19.91},"min":{"1min":6.8529999999999998,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":166.327,"5min":166.327,"15min":270.91399999999999},"last":8.4440000000000008},{"interface":"front","average":{"1min":24.475999999999999,"5min":17.946000000000002,"15min":24.292999999999999},"min":{"1min":13.491,"5min":10.361000000000001,"15min":8.8279999999999994},"max":{"1min":130.548,"5min":130.548,"15min":259.60399999999998},"last":17.736000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:04:51 2024","interfaces":[{"interface":"back","average":{"1min":17.291,"5min":17.452999999999999,"15min":26.175000000000001},"min":{"1min":12.855,"5min":11.922000000000001,"15min":7.9800000000000004},"max":{"1min":20.937000000000001,"5min":31.047000000000001,"15min":321.79500000000002},"last":19.460000000000001},{"interface":"front","average":{"1min":12.227,"5min":12.388,"15min":21.454999999999998},"min":{"1min":8.5169999999999995,"5min":7.9699999999999998,"15min":7.7229999999999999},"max":{"1min":20.093,"5min":28.638999999999999,"15min":282.54599999999999},"last":12.933}]},{"osd":7,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":24.565000000000001,"5min":19.756,"15min":26.001000000000001},"min":{"1min":16.887,"5min":13.028,"15min":8.8460000000000001},"max":{"1min":67.106999999999999,"5min":67.106999999999999,"15min":312.435},"last":18.59},{"interface":"front","average":{"1min":20.120000000000001,"5min":15.542,"15min":23.835000000000001},"min":{"1min":8.5540000000000003,"5min":8.2370000000000001,"15min":8.2370000000000001},"max":{"1min":97.379000000000005,"5min":97.379000000000005,"15min":256.11900000000003},"last":14.731999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509825,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74864,"kb_used_data":11752,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94296976,"statfs":{"total":96636764160,"available":96560103424,"internally_reserved":0,"allocated":12034048,"data_stored":6756061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:04:45 2024","interfaces":[{"interface":"back","average":{"1min":8.1910000000000007,"5min":8.7469999999999999,"15min":16.152999999999999},"min":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"max":{"1min":17.446000000000002,"5min":27.192,"15min":209.96299999999999},"last":7.2469999999999999},{"interface":"front","average":{"1min":9.2430000000000003,"5min":9.1839999999999993,"15min":17.379000000000001},"min":{"1min":5.3650000000000002,"5min":4.9560000000000004,"15min":4.7549999999999999},"max":{"1min":14.426,"5min":25.323,"15min":244.054},"last":8.2629999999999999}]},{"osd":2,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":25.413,"5min":13.148999999999999,"15min":17.792999999999999},"min":{"1min":8.1020000000000003,"5min":6.202,"15min":4.8819999999999997},"max":{"1min":218.80500000000001,"5min":218.80500000000001,"15min":221.54300000000001},"last":8.0009999999999994},{"interface":"front","average":{"1min":24.704999999999998,"5min":15.188000000000001,"15min":20.256},"min":{"1min":10.317,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":144.12899999999999,"5min":144.12899999999999,"15min":219.34899999999999},"last":13.866}]},{"osd":3,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":14.996,"5min":13.903,"15min":21.224},"min":{"1min":7.3200000000000003,"5min":6.569,"15min":6.569},"max":{"1min":40.685000000000002,"5min":40.685000000000002,"15min":163.58799999999999},"last":17.402000000000001},{"interface":"front","average":{"1min":14.272,"5min":12.506,"15min":19.687999999999999},"min":{"1min":7.9630000000000001,"5min":7.0549999999999997,"15min":6.4809999999999999},"max":{"1min":55.024999999999999,"5min":55.024999999999999,"15min":244.178},"last":15.630000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:04:45 2024","interfaces":[{"interface":"back","average":{"1min":15.436999999999999,"5min":15.552,"15min":21.210000000000001},"min":{"1min":7.6369999999999996,"5min":7.6369999999999996,"15min":7.6369999999999996},"max":{"1min":19.053000000000001,"5min":23.524999999999999,"15min":241.94800000000001},"last":18.265000000000001},{"interface":"front","average":{"1min":10.997,"5min":10.789999999999999,"15min":18.545000000000002},"min":{"1min":7.0789999999999997,"5min":6.7069999999999999,"15min":6.7069999999999999},"max":{"1min":18.260999999999999,"5min":18.260999999999999,"15min":225.60499999999999},"last":12.446999999999999}]},{"osd":5,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":26.631,"5min":16.741,"15min":22.994},"min":{"1min":9.1850000000000005,"5min":9.1850000000000005,"15min":6.8970000000000002},"max":{"1min":156.554,"5min":156.554,"15min":313.36599999999999},"last":10.292999999999999},{"interface":"front","average":{"1min":26.550000000000001,"5min":16.300999999999998,"15min":23.606000000000002},"min":{"1min":8.9369999999999994,"5min":8.9369999999999994,"15min":5.851},"max":{"1min":138.114,"5min":138.114,"15min":298.49599999999998},"last":9.0289999999999999}]},{"osd":6,"last update":"Sun Jan 21 05:04:45 2024","interfaces":[{"interface":"back","average":{"1min":15.303000000000001,"5min":15.319000000000001,"15min":24.856000000000002},"min":{"1min":9.827,"5min":8.8949999999999996,"15min":8.8949999999999996},"max":{"1min":19.283000000000001,"5min":23.902999999999999,"15min":298.51999999999998},"last":11.117000000000001},{"interface":"front","average":{"1min":16.611999999999998,"5min":16.690999999999999,"15min":24.472999999999999},"min":{"1min":11.101000000000001,"5min":10.550000000000001,"15min":7.4880000000000004},"max":{"1min":21.846,"5min":26.234000000000002,"15min":181.01599999999999},"last":13.145}]},{"osd":7,"last update":"Sun Jan 21 05:05:17 2024","interfaces":[{"interface":"back","average":{"1min":21.521000000000001,"5min":16.407,"15min":24.297999999999998},"min":{"1min":11.332000000000001,"5min":9.8070000000000004,"15min":9.3230000000000004},"max":{"1min":107.875,"5min":107.875,"15min":214.24199999999999},"last":11.731},{"interface":"front","average":{"1min":20.009,"5min":17.219999999999999,"15min":24.443000000000001},"min":{"1min":9.5779999999999994,"5min":9.5779999999999994,"15min":9.4120000000000008},"max":{"1min":51.546999999999997,"5min":51.546999999999997,"15min":180.73099999999999},"last":16.562999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575233,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74860,"kb_used_data":11748,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94296980,"statfs":{"total":96636764160,"available":96560107520,"internally_reserved":0,"allocated":12029952,"data_stored":6756056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":23.058,"5min":12.195,"15min":18.460000000000001},"min":{"1min":6.226,"5min":5.2300000000000004,"15min":4.601},"max":{"1min":154.13399999999999,"5min":154.13399999999999,"15min":377.50999999999999},"last":13.148},{"interface":"front","average":{"1min":32.405999999999999,"5min":17.396999999999998,"15min":21.706},"min":{"1min":7.8029999999999999,"5min":5.8559999999999999,"15min":4.4359999999999999},"max":{"1min":193.01400000000001,"5min":193.01400000000001,"15min":224.58600000000001},"last":22.873999999999999}]},{"osd":1,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":24.754999999999999,"5min":14.738,"15min":22.327999999999999},"min":{"1min":6.165,"5min":5.6820000000000004,"15min":4.5380000000000003},"max":{"1min":160.279,"5min":160.279,"15min":368.79199999999997},"last":10.007},{"interface":"front","average":{"1min":28.187000000000001,"5min":13.547000000000001,"15min":19.158999999999999},"min":{"1min":9.9169999999999998,"5min":4.944,"15min":4.3170000000000002},"max":{"1min":156.38499999999999,"5min":156.38499999999999,"15min":225.21000000000001},"last":7.5839999999999996}]},{"osd":2,"last update":"Sun Jan 21 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":28.533000000000001,"5min":14.004,"15min":20.603999999999999},"min":{"1min":7.7069999999999999,"5min":6.6420000000000003,"15min":5.8460000000000001},"max":{"1min":232.78200000000001,"5min":232.78200000000001,"15min":232.78200000000001},"last":11.324999999999999},{"interface":"front","average":{"1min":35.103000000000002,"5min":15.885999999999999,"15min":22.988},"min":{"1min":7.9740000000000002,"5min":6.4199999999999999,"15min":5.8289999999999997},"max":{"1min":288.49099999999999,"5min":288.49099999999999,"15min":309.24599999999998},"last":6.6349999999999998}]},{"osd":3,"last update":"Sun Jan 21 05:04:44 2024","interfaces":[{"interface":"back","average":{"1min":15.166,"5min":15.417,"15min":25.603000000000002},"min":{"1min":11.159000000000001,"5min":8.516,"15min":6.7229999999999999},"max":{"1min":18.838000000000001,"5min":31.256,"15min":375.07799999999997},"last":12.566000000000001},{"interface":"front","average":{"1min":12.089,"5min":11.997999999999999,"15min":22.629999999999999},"min":{"1min":7.6299999999999999,"5min":6.4589999999999996,"15min":6.4589999999999996},"max":{"1min":16.795999999999999,"5min":32.795999999999999,"15min":276.57100000000003},"last":14.372}]},{"osd":4,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":31.202999999999999,"5min":15.868,"15min":24.077000000000002},"min":{"1min":8.9039999999999999,"5min":6.3360000000000003,"15min":6.3360000000000003},"max":{"1min":184.767,"5min":184.767,"15min":272.274},"last":8.3659999999999997},{"interface":"front","average":{"1min":33.640999999999998,"5min":20.071999999999999,"15min":28.475000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":6.8689999999999998},"max":{"1min":204.47499999999999,"5min":204.47499999999999,"15min":378.404},"last":21.052}]},{"osd":6,"last update":"Sun Jan 21 05:04:44 2024","interfaces":[{"interface":"back","average":{"1min":13.416,"5min":12.768000000000001,"15min":23.533999999999999},"min":{"1min":9.0690000000000008,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":19.172000000000001,"5min":32.139000000000003,"15min":296.88499999999999},"last":17.539999999999999},{"interface":"front","average":{"1min":16.545999999999999,"5min":16.481999999999999,"15min":30.184999999999999},"min":{"1min":11.481999999999999,"5min":10.188000000000001,"15min":9.6750000000000007},"max":{"1min":20.914000000000001,"5min":37.082999999999998,"15min":311.10899999999998},"last":18.202999999999999}]},{"osd":7,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":31.890000000000001,"5min":20.614999999999998,"15min":29.431999999999999},"min":{"1min":10.763,"5min":9.9640000000000004,"15min":8.032},"max":{"1min":202.90700000000001,"5min":202.90700000000001,"15min":486.358},"last":21.878},{"interface":"front","average":{"1min":36.167000000000002,"5min":20.385999999999999,"15min":27.478999999999999},"min":{"1min":10.587999999999999,"5min":9.8740000000000006,"15min":9.1039999999999992},"max":{"1min":177.25800000000001,"5min":177.25800000000001,"15min":486.62900000000002},"last":18.789000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T05:05:45.785 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-21T05:05:45.785 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.883+0000 7f5fb8b89640 10 monclient: get_monmap_and_config 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.883+0000 7f5fb8b89640 10 monclient: build_initial_monmap 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.883+0000 7f5fb8b89640 10 monclient: monmap: 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:45.884186+0000 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:45.884186+0000 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:45.885 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.883+0000 7f5fb8b89640 1 Processor -- start 2024-01-21T05:05:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.883+0000 7f5fb8b89640 1 -- start start 2024-01-21T05:05:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.883+0000 7f5fb8b89640 10 monclient: init 2024-01-21T05:05:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-01-21T05:05:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5fb40aee30 0x7f5fb40721e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 10 monclient(hunting): picked mon.noname-a con 0x7f5fb40aee30 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 0x7f5fb4142290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 10 monclient(hunting): picked mon.noname-b con 0x7f5fb4141e30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5fb413b320 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb41427d0 con 0x7f5fb413b320 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb41419e0 con 0x7f5fb40aee30 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40af210 con 0x7f5fb4141e30 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.884+0000 7f5fb8b89640 10 monclient(hunting): authenticate will time out at 3304.225865s 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb259c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5fb40aee30 0x7f5fb40721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb1d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 0x7f5fb4142290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb1d9b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 0x7f5fb4142290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:48498/0 (socket says 172.21.15.69:48498) 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb1d9b640 1 -- 172.21.15.69:0/1902859121 learned_addr learned my addr 172.21.15.69:0/1902859121 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb1d9b640 10 monclient(hunting): get_auth_request con 0x7f5fb4141e30 auth_method 0 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb1d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:45.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:45.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb259c640 10 monclient(hunting): get_auth_request con 0x7f5fb40aee30 auth_method 0 2024-01-21T05:05:45.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:45.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb259c640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:45.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.885+0000 7f5fb259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:45.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.886+0000 7f5fb1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:45.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.886+0000 7f5fb1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:45.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.886+0000 7f5fb1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:45.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.886+0000 7f5fb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:45.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.886+0000 7f5fb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:45.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.886+0000 7f5fb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:45.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.887+0000 7f5fb259c640 10 monclient(hunting): handle_auth_done global_id 17424 payload 274 2024-01-21T05:05:45.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.887+0000 7f5fb259c640 1 -- 172.21.15.69:0/1902859121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 msgr2=0x7f5fb4142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:45.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.887+0000 7f5fb259c640 1 --2- 172.21.15.69:0/1902859121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 0x7f5fb4142290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.887+0000 7f5fb259c640 1 -- 172.21.15.69:0/1902859121 >> v1:172.21.15.69:6789/0 conn(0x7f5fb413b320 legacy=0x7f5fb413b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:45.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.887+0000 7f5fb259c640 10 monclient: _finish_hunting 0 2024-01-21T05:05:45.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.887+0000 7f5fb259c640 1 monclient: found mon.noname-a 2024-01-21T05:05:45.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.887+0000 7f5fb259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:45.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.887+0000 7f5fb259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:45.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.887+0000 7f5fb259c640 1 -- 172.21.15.69:0/1902859121 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb4072720 con 0x7f5fb40aee30 2024-01-21T05:05:45.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.887+0000 7f5fb1d9b640 1 --2- 172.21.15.69:0/1902859121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 0x7f5fb4142290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T05:05:45.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.889+0000 7f5fb259c640 1 --2- 172.21.15.69:0/1902859121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fb40aee30 0x7f5fb40721e0 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4141b20 tx=0x7f5fa402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be29596220bafac server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 1 -- 172.21.15.69:0/1902859121 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4031800 con 0x7f5fb40aee30 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 10 monclient: dump: 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:45.895 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 10 monclient: _finish_auth 0 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 10 monclient: _check_auth_tickets 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 1 -- 172.21.15.69:0/1902859121 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa4031960 con 0x7f5fb40aee30 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb8b89640 5 monclient: authenticate success, global_id 17424 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb8b89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb8b89640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 1 -- 172.21.15.69:0/1902859121 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4031cf0 con 0x7f5fb40aee30 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb8b89640 4 set_mon_vals no callback set 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb159a640 10 monclient: dump: 2024-01-21T05:05:45.896 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:45.897 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:45.897 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.897 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.897 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:45.897 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.897 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:45.897 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:45.897 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:45.897 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb8b89640 10 monclient: shutdown 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb8b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb8b89640 1 -- 172.21.15.69:0/1902859121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fb40aee30 msgr2=0x7f5fb40721e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb8b89640 1 --2- 172.21.15.69:0/1902859121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fb40aee30 0x7f5fb40721e0 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4141b20 tx=0x7f5fa402b8a0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb8b89640 1 -- 172.21.15.69:0/1902859121 shutdown_connections 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb8b89640 1 --2- 172.21.15.69:0/1902859121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 0x7f5fb4142290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.894+0000 7f5fb8b89640 1 --2- 172.21.15.69:0/1902859121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fb40aee30 0x7f5fb40721e0 unknown :-1 s=CLOSED pgs=3486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.895+0000 7f5fb8b89640 1 -- 172.21.15.69:0/1902859121 >> 172.21.15.69:0/1902859121 conn(0x7f5fb40b3960 msgr2=0x7f5fb40b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.895+0000 7f5fb8b89640 1 -- 172.21.15.69:0/1902859121 shutdown_connections 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.895+0000 7f5fb8b89640 1 -- 172.21.15.69:0/1902859121 wait complete. 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.895+0000 7f5fb8b89640 10 monclient: build_initial_monmap 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.895+0000 7f5fb8b89640 10 monclient: monmap: 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:45.901 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 librados: starting msgr at 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 librados: starting objecter 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 Processor -- start 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 -- start start 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 librados: setting wanted keys 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 librados: calling monclient init 2024-01-21T05:05:45.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 10 monclient: init 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5fb40aee30 0x7f5fb406f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 10 monclient(hunting): picked mon.noname-b con 0x7f5fb40aee30 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fb413b320 0x7f5fb4139750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 10 monclient(hunting): picked mon.noname-a con 0x7f5fb413b320 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 0x7f5fb4139c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 10 monclient(hunting): picked mon.noname-c con 0x7f5fb4141e30 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40b2fd0 con 0x7f5fb413b320 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb41c9700 con 0x7f5fb40aee30 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb41c92e0 con 0x7f5fb4141e30 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.896+0000 7f5fb8b89640 10 monclient(hunting): authenticate will time out at 3304.238198s 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb1d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fb413b320 0x7f5fb4139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb2d9d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 0x7f5fb4139c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb1d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fb413b320 0x7f5fb4139750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41572/0 (socket says 172.21.15.69:41572) 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb259c640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5fb40aee30 0x7f5fb406f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb1d9b640 1 -- 172.21.15.69:0/3562481025 learned_addr learned my addr 172.21.15.69:0/3562481025 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb1d9b640 10 monclient(hunting): get_auth_request con 0x7f5fb413b320 auth_method 0 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb1d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb2d9d640 10 monclient(hunting): get_auth_request con 0x7f5fb4141e30 auth_method 0 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb2d9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.897+0000 7f5fb2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb259c640 10 monclient(hunting): get_auth_request con 0x7f5fb40aee30 auth_method 0 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb259c640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb259c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:45.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.898+0000 7f5fb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:45.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.901+0000 7f5fb1d9b640 10 monclient(hunting): handle_auth_done global_id 17427 payload 824 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.901+0000 7f5fb1d9b640 1 -- 172.21.15.69:0/3562481025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 msgr2=0x7f5fb4139c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.901+0000 7f5fb1d9b640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 0x7f5fb4139c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.901+0000 7f5fb1d9b640 1 -- 172.21.15.69:0/3562481025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5fb40aee30 msgr2=0x7f5fb406f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.901+0000 7f5fb1d9b640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5fb40aee30 0x7f5fb406f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.901+0000 7f5fb1d9b640 10 monclient: _finish_hunting 0 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.901+0000 7f5fb1d9b640 1 monclient: found mon.noname-a 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.901+0000 7f5fb1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.901+0000 7f5fb1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.901+0000 7f5fb1d9b640 1 -- 172.21.15.69:0/3562481025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa80094d0 con 0x7f5fb413b320 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.901+0000 7f5fb259c640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5fb40aee30 0x7f5fb406f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.903+0000 7f5fb1d9b640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fb413b320 0x7f5fb4139750 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800e9e0 tx=0x7f5fa800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.903+0000 7f5fa2ffd640 10 client.?.objecter ms_handle_connect 0x7f5fb413b320 2024-01-21T05:05:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.903+0000 7f5fa2ffd640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:45.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 1 -- 172.21.15.69:0/3562481025 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa800c9d0 con 0x7f5fb413b320 2024-01-21T05:05:45.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:45.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:45.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 10 monclient: dump: 2024-01-21T05:05:45.910 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:45.910 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:45.910 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.910 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.910 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:45.910 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 10 monclient: _finish_auth 0 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 10 monclient: _check_auth_tickets 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 1 -- 172.21.15.69:0/3562481025 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa800cb30 con 0x7f5fb413b320 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 1 -- 172.21.15.69:0/3562481025 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa800cea0 con 0x7f5fb413b320 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fb8b89640 5 monclient: authenticate success, global_id 17427 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fb8b89640 10 monclient: _renew_subs 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fb8b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fb8b89640 1 -- 172.21.15.69:0/3562481025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa4009400 con 0x7f5fb413b320 2024-01-21T05:05:45.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fb0d99640 4 set_mon_vals no callback set 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fa2ffd640 10 monclient: dump: 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:45.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fb8b89640 10 client.17427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:45.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fb8b89640 10 monclient: _renew_subs 2024-01-21T05:05:45.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fb8b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:45.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fb8b89640 1 -- 172.21.15.69:0/3562481025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb40b2a20 con 0x7f5fb413b320 2024-01-21T05:05:45.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.909+0000 7f5fb8b89640 1 librados: init done 2024-01-21T05:05:45.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.910+0000 7f5fb8b89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:45.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.910+0000 7f5fb8b89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:45.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.910+0000 7f5fb8b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:45.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.910+0000 7f5fb8b89640 1 -- 172.21.15.69:0/3562481025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb40b3180 con 0x7f5fb413b320 2024-01-21T05:05:45.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.928+0000 7f5fa2ffd640 1 -- 172.21.15.69:0/3562481025 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f5fa8013470 con 0x7f5fb413b320 2024-01-21T05:05:45.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.928+0000 7f5fa2ffd640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5f9406c8e0 0x7f5f9406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:45.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.929+0000 7f5fb259c640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5f9406c8e0 0x7f5f9406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:45.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.929+0000 7f5fb259c640 10 monclient: get_auth_request con 0x7f5f9406c8e0 auth_method 0 2024-01-21T05:05:45.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.929+0000 7f5fb259c640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5f9406c8e0 0x7f5f9406eda0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4036000 tx=0x7f5fa403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:45.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.929+0000 7f5fa2ffd640 10 client.17427.objecter ms_handle_connect 0x7f5f9406c8e0 2024-01-21T05:05:45.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.931+0000 7f5fa2ffd640 1 -- 172.21.15.69:0/3562481025 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1109..1109 src has 1032..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f5fa8058bc0 con 0x7f5fb413b320 2024-01-21T05:05:45.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.931+0000 7f5fa2ffd640 10 client.17427.objecter ms_dispatch 0x7f5fb4000b90 osd_map(1109..1109 src has 1032..1109) v4 2024-01-21T05:05:45.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.931+0000 7f5fa2ffd640 3 client.17427.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:45.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.931+0000 7f5fa2ffd640 3 client.17427.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:45.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:45.931+0000 7f5fa2ffd640 20 client.17427.objecter dump_active .. 0 homeless 2024-01-21T05:05:46.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.168+0000 7f5fa2ffd640 1 -- 172.21.15.69:0/3562481025 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f5fa8054f00 con 0x7f5fb413b320 2024-01-21T05:05:46.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.168+0000 7f5fa2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:46.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.168+0000 7f5fa2ffd640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:46.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.344+0000 7f5fb8b89640 1 -- 172.21.15.69:0/3562481025 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5fb41c9450 con 0x7f5f9406c8e0 2024-01-21T05:05:46.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.349+0000 7f5fa2ffd640 1 -- 172.21.15.69:0/3562481025 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151755 (secure 0 0 0) 0x7f5fb41c9450 con 0x7f5f9406c8e0 2024-01-21T05:05:46.351 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T05:05:46.352 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 10 librados: watch_flush enter 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 10 librados: watch_flush exit 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 20 client.17427.objecter shutdown clearing up homeless session... 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 10 client.17427.objecter successfully canceled tick 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 1 -- 172.21.15.69:0/3562481025 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5f9406c8e0 msgr2=0x7f5f9406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5f9406c8e0 0x7f5f9406eda0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4036000 tx=0x7f5fa403c040 comp rx=0 tx=0).stop 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 10 monclient: shutdown 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 1 -- 172.21.15.69:0/3562481025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fb413b320 msgr2=0x7f5fb4139750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fb413b320 0x7f5fb4139750 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800e9e0 tx=0x7f5fa800eea0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 1 -- 172.21.15.69:0/3562481025 shutdown_connections 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f5f9406c8e0 0x7f5f9406eda0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f5fb4141e30 0x7f5fb4139c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fb413b320 0x7f5fb4139750 unknown :-1 s=CLOSED pgs=3487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 1 --2- 172.21.15.69:0/3562481025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5fb40aee30 0x7f5fb406f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 1 -- 172.21.15.69:0/3562481025 >> 172.21.15.69:0/3562481025 conn(0x7f5fb40b3960 msgr2=0x7f5fb40b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:46.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.351+0000 7f5fb8b89640 1 -- 172.21.15.69:0/3562481025 shutdown_connections 2024-01-21T05:05:46.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.352+0000 7f5fb8b89640 1 -- 172.21.15.69:0/3562481025 wait complete. 2024-01-21T05:05:46.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.352+0000 7f5fb8b89640 1 librados: shutdown 2024-01-21T05:05:46.365 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":2080,"stamp":"2024-01-21T05:05:46.034131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":609108,"kb_used_data":90756,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754365612,"statfs":{"total":773094113280,"available":772470386688,"internally_reserved":0,"allocated":92934144,"data_stored":51061628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":529714736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":85,"apply_latency_ms":85,"commit_latency_ns":85000000,"apply_latency_ns":85000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-3,"num_read_kb":-3,"num_write":-2,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001456"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":1046,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.057494+0000","last_change":"2024-01-21T04:34:59.893360+0000","last_active":"2024-01-21T05:05:25.057494+0000","last_peered":"2024-01-21T05:05:25.057494+0000","last_clean":"2024-01-21T05:05:25.057494+0000","last_became_active":"2024-01-21T04:34:59.874210+0000","last_became_peered":"2024-01-21T04:34:59.874210+0000","last_unstale":"2024-01-21T05:05:25.057494+0000","last_undegraded":"2024-01-21T05:05:25.057494+0000","last_fullsized":"2024-01-21T05:05:25.057494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:03:39.790631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1063,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.109554+0000","last_change":"2024-01-21T04:35:19.549398+0000","last_active":"2024-01-21T05:05:25.109554+0000","last_peered":"2024-01-21T05:05:25.109554+0000","last_clean":"2024-01-21T05:05:25.109554+0000","last_became_active":"2024-01-21T04:35:19.542355+0000","last_became_peered":"2024-01-21T04:35:19.542355+0000","last_unstale":"2024-01-21T05:05:25.109554+0000","last_undegraded":"2024-01-21T05:05:25.109554+0000","last_fullsized":"2024-01-21T05:05:25.109554+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:17:21.163426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1085,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.257683+0000","last_change":"2024-01-21T04:35:21.254053+0000","last_active":"2024-01-21T05:05:25.257683+0000","last_peered":"2024-01-21T05:05:25.257683+0000","last_clean":"2024-01-21T05:05:25.257683+0000","last_became_active":"2024-01-21T04:35:21.234113+0000","last_became_peered":"2024-01-21T04:35:21.234113+0000","last_unstale":"2024-01-21T05:05:25.257683+0000","last_undegraded":"2024-01-21T05:05:25.257683+0000","last_fullsized":"2024-01-21T05:05:25.257683+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:52:02.299953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1050,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.013476+0000","last_change":"2024-01-21T04:35:23.469523+0000","last_active":"2024-01-21T05:05:25.013476+0000","last_peered":"2024-01-21T05:05:25.013476+0000","last_clean":"2024-01-21T05:05:25.013476+0000","last_became_active":"2024-01-21T04:35:23.463341+0000","last_became_peered":"2024-01-21T04:35:23.463341+0000","last_unstale":"2024-01-21T05:05:25.013476+0000","last_undegraded":"2024-01-21T05:05:25.013476+0000","last_fullsized":"2024-01-21T05:05:25.013476+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:40:34.277268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1051,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.976830+0000","last_change":"2024-01-21T04:35:16.948777+0000","last_active":"2024-01-21T05:05:25.976830+0000","last_peered":"2024-01-21T05:05:25.976830+0000","last_clean":"2024-01-21T05:05:25.976830+0000","last_became_active":"2024-01-21T04:35:16.942042+0000","last_became_peered":"2024-01-21T04:35:16.942042+0000","last_unstale":"2024-01-21T05:05:25.976830+0000","last_undegraded":"2024-01-21T05:05:25.976830+0000","last_fullsized":"2024-01-21T05:05:25.976830+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:15:32.933069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"116'7","reported_seq":1077,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.918924+0000","last_change":"2024-01-21T04:35:25.379657+0000","last_active":"2024-01-21T05:05:24.918924+0000","last_peered":"2024-01-21T05:05:24.918924+0000","last_clean":"2024-01-21T05:05:24.918924+0000","last_became_active":"2024-01-21T04:35:25.364561+0000","last_became_peered":"2024-01-21T04:35:25.364561+0000","last_unstale":"2024-01-21T05:05:24.918924+0000","last_undegraded":"2024-01-21T05:05:24.918924+0000","last_fullsized":"2024-01-21T05:05:24.918924+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:49:49.587513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1084,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.251316+0000","last_change":"2024-01-21T04:34:53.473895+0000","last_active":"2024-01-21T05:05:25.251316+0000","last_peered":"2024-01-21T05:05:25.251316+0000","last_clean":"2024-01-21T05:05:25.251316+0000","last_became_active":"2024-01-21T04:34:53.465163+0000","last_became_peered":"2024-01-21T04:34:53.465163+0000","last_unstale":"2024-01-21T05:05:25.251316+0000","last_undegraded":"2024-01-21T05:05:25.251316+0000","last_fullsized":"2024-01-21T05:05:25.251316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:37:17.441183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1050,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:43.832568+0000","last_change":"2024-01-21T04:35:02.604107+0000","last_active":"2024-01-21T05:05:43.832568+0000","last_peered":"2024-01-21T05:05:43.832568+0000","last_clean":"2024-01-21T05:05:43.832568+0000","last_became_active":"2024-01-21T04:35:02.595993+0000","last_became_peered":"2024-01-21T04:35:02.595993+0000","last_unstale":"2024-01-21T05:05:43.832568+0000","last_undegraded":"2024-01-21T05:05:43.832568+0000","last_fullsized":"2024-01-21T05:05:43.832568+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:05:40.627686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1084,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.237542+0000","last_change":"2024-01-21T04:35:21.226772+0000","last_active":"2024-01-21T05:05:25.237542+0000","last_peered":"2024-01-21T05:05:25.237542+0000","last_clean":"2024-01-21T05:05:25.237542+0000","last_became_active":"2024-01-21T04:35:21.218816+0000","last_became_peered":"2024-01-21T04:35:21.218816+0000","last_unstale":"2024-01-21T05:05:25.237542+0000","last_undegraded":"2024-01-21T05:05:25.237542+0000","last_fullsized":"2024-01-21T05:05:25.237542+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:07:02.731410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1049,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.132874+0000","last_change":"2024-01-21T04:34:53.062240+0000","last_active":"2024-01-21T05:05:26.132874+0000","last_peered":"2024-01-21T05:05:26.132874+0000","last_clean":"2024-01-21T05:05:26.132874+0000","last_became_active":"2024-01-21T04:34:53.056955+0000","last_became_peered":"2024-01-21T04:34:53.056955+0000","last_unstale":"2024-01-21T05:05:26.132874+0000","last_undegraded":"2024-01-21T05:05:26.132874+0000","last_fullsized":"2024-01-21T05:05:26.132874+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:03:25.878354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1065,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.081092+0000","last_change":"2024-01-21T04:35:00.848090+0000","last_active":"2024-01-21T05:05:25.081092+0000","last_peered":"2024-01-21T05:05:25.081092+0000","last_clean":"2024-01-21T05:05:25.081092+0000","last_became_active":"2024-01-21T04:35:00.838638+0000","last_became_peered":"2024-01-21T04:35:00.838638+0000","last_unstale":"2024-01-21T05:05:25.081092+0000","last_undegraded":"2024-01-21T05:05:25.081092+0000","last_fullsized":"2024-01-21T05:05:25.081092+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:25:11.256572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1047,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:27.034452+0000","last_change":"2024-01-21T04:35:28.444122+0000","last_active":"2024-01-21T05:05:27.034452+0000","last_peered":"2024-01-21T05:05:27.034452+0000","last_clean":"2024-01-21T05:05:27.034452+0000","last_became_active":"2024-01-21T04:35:28.438809+0000","last_became_peered":"2024-01-21T04:35:28.438809+0000","last_unstale":"2024-01-21T05:05:27.034452+0000","last_undegraded":"2024-01-21T05:05:27.034452+0000","last_fullsized":"2024-01-21T05:05:27.034452+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:27:38.339736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1049,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.984819+0000","last_change":"2024-01-21T04:35:00.835300+0000","last_active":"2024-01-21T05:05:24.984819+0000","last_peered":"2024-01-21T05:05:24.984819+0000","last_clean":"2024-01-21T05:05:24.984819+0000","last_became_active":"2024-01-21T04:35:00.824479+0000","last_became_peered":"2024-01-21T04:35:00.824479+0000","last_unstale":"2024-01-21T05:05:24.984819+0000","last_undegraded":"2024-01-21T05:05:24.984819+0000","last_fullsized":"2024-01-21T05:05:24.984819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:11:26.198997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":1179,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.909621+0000","last_change":"2024-01-21T04:33:33.164364+0000","last_active":"2024-01-21T05:05:24.909621+0000","last_peered":"2024-01-21T05:05:24.909621+0000","last_clean":"2024-01-21T05:05:24.909621+0000","last_became_active":"2024-01-21T04:33:33.133732+0000","last_became_peered":"2024-01-21T04:33:33.133732+0000","last_unstale":"2024-01-21T05:05:24.909621+0000","last_undegraded":"2024-01-21T05:05:24.909621+0000","last_fullsized":"2024-01-21T05:05:24.909621+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:29.982915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:29.982915+0000","last_clean_scrub_stamp":"2024-01-21T04:33:29.982915+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:02:00.259080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1057,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.958852+0000","last_change":"2024-01-21T04:33:48.369523+0000","last_active":"2024-01-21T05:05:24.958852+0000","last_peered":"2024-01-21T05:05:24.958852+0000","last_clean":"2024-01-21T05:05:24.958852+0000","last_became_active":"2024-01-21T04:33:48.364387+0000","last_became_peered":"2024-01-21T04:33:48.364387+0000","last_unstale":"2024-01-21T05:05:24.958852+0000","last_undegraded":"2024-01-21T05:05:24.958852+0000","last_fullsized":"2024-01-21T05:05:24.958852+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:06:07.142172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1047,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:27.060750+0000","last_change":"2024-01-21T04:34:54.460251+0000","last_active":"2024-01-21T05:05:27.060750+0000","last_peered":"2024-01-21T05:05:27.060750+0000","last_clean":"2024-01-21T05:05:27.060750+0000","last_became_active":"2024-01-21T04:34:54.452154+0000","last_became_peered":"2024-01-21T04:34:54.452154+0000","last_unstale":"2024-01-21T05:05:27.060750+0000","last_undegraded":"2024-01-21T05:05:27.060750+0000","last_fullsized":"2024-01-21T05:05:27.060750+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:03:09.023273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1057,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.888382+0000","last_change":"2024-01-21T04:33:47.762316+0000","last_active":"2024-01-21T05:05:24.888382+0000","last_peered":"2024-01-21T05:05:24.888382+0000","last_clean":"2024-01-21T05:05:24.888382+0000","last_became_active":"2024-01-21T04:33:47.734733+0000","last_became_peered":"2024-01-21T04:33:47.734733+0000","last_unstale":"2024-01-21T05:05:24.888382+0000","last_undegraded":"2024-01-21T05:05:24.888382+0000","last_fullsized":"2024-01-21T05:05:24.888382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:16:54.133000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1049,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.016911+0000","last_change":"2024-01-21T04:35:05.297742+0000","last_active":"2024-01-21T05:05:25.016911+0000","last_peered":"2024-01-21T05:05:25.016911+0000","last_clean":"2024-01-21T05:05:25.016911+0000","last_became_active":"2024-01-21T04:35:05.290888+0000","last_became_peered":"2024-01-21T04:35:05.290888+0000","last_unstale":"2024-01-21T05:05:25.016911+0000","last_undegraded":"2024-01-21T05:05:25.016911+0000","last_fullsized":"2024-01-21T05:05:25.016911+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:51:16.216402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1054,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.951406+0000","last_change":"2024-01-21T04:33:48.697481+0000","last_active":"2024-01-21T05:05:26.951406+0000","last_peered":"2024-01-21T05:05:26.951406+0000","last_clean":"2024-01-21T05:05:26.951406+0000","last_became_active":"2024-01-21T04:33:48.670864+0000","last_became_peered":"2024-01-21T04:33:48.670864+0000","last_unstale":"2024-01-21T05:05:26.951406+0000","last_undegraded":"2024-01-21T05:05:26.951406+0000","last_fullsized":"2024-01-21T05:05:26.951406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:00:07.393617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1047,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:27.017847+0000","last_change":"2024-01-21T04:35:07.348347+0000","last_active":"2024-01-21T05:05:27.017847+0000","last_peered":"2024-01-21T05:05:27.017847+0000","last_clean":"2024-01-21T05:05:27.017847+0000","last_became_active":"2024-01-21T04:35:07.340277+0000","last_became_peered":"2024-01-21T04:35:07.340277+0000","last_unstale":"2024-01-21T05:05:27.017847+0000","last_undegraded":"2024-01-21T05:05:27.017847+0000","last_fullsized":"2024-01-21T05:05:27.017847+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:33:05.933710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1059,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.943953+0000","last_change":"2024-01-21T04:37:13.894971+0000","last_active":"2024-01-21T05:05:24.943953+0000","last_peered":"2024-01-21T05:05:24.943953+0000","last_clean":"2024-01-21T05:05:24.943953+0000","last_became_active":"2024-01-21T04:33:48.015845+0000","last_became_peered":"2024-01-21T04:33:48.015845+0000","last_unstale":"2024-01-21T05:05:24.943953+0000","last_undegraded":"2024-01-21T05:05:24.943953+0000","last_fullsized":"2024-01-21T05:05:24.943953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:42:22.571268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1050,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:43.836852+0000","last_change":"2024-01-21T04:35:16.132218+0000","last_active":"2024-01-21T05:05:43.836852+0000","last_peered":"2024-01-21T05:05:43.836852+0000","last_clean":"2024-01-21T05:05:43.836852+0000","last_became_active":"2024-01-21T04:35:16.127034+0000","last_became_peered":"2024-01-21T04:35:16.127034+0000","last_unstale":"2024-01-21T05:05:43.836852+0000","last_undegraded":"2024-01-21T05:05:43.836852+0000","last_fullsized":"2024-01-21T05:05:43.836852+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:29:27.366404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1054,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.059062+0000","last_change":"2024-01-21T04:33:48.068776+0000","last_active":"2024-01-21T05:05:25.059062+0000","last_peered":"2024-01-21T05:05:25.059062+0000","last_clean":"2024-01-21T05:05:25.059062+0000","last_became_active":"2024-01-21T04:33:48.032152+0000","last_became_peered":"2024-01-21T04:33:48.032152+0000","last_unstale":"2024-01-21T05:05:25.059062+0000","last_undegraded":"2024-01-21T05:05:25.059062+0000","last_fullsized":"2024-01-21T05:05:25.059062+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:12:43.256934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1047,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.957819+0000","last_change":"2024-01-21T04:34:55.542633+0000","last_active":"2024-01-21T05:05:26.957819+0000","last_peered":"2024-01-21T05:05:26.957819+0000","last_clean":"2024-01-21T05:05:26.957819+0000","last_became_active":"2024-01-21T04:34:55.527971+0000","last_became_peered":"2024-01-21T04:34:55.527971+0000","last_unstale":"2024-01-21T05:05:26.957819+0000","last_undegraded":"2024-01-21T05:05:26.957819+0000","last_fullsized":"2024-01-21T05:05:26.957819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:19:42.501825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1056,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.986390+0000","last_change":"2024-01-21T04:33:47.840628+0000","last_active":"2024-01-21T05:05:24.986390+0000","last_peered":"2024-01-21T05:05:24.986390+0000","last_clean":"2024-01-21T05:05:24.986390+0000","last_became_active":"2024-01-21T04:33:47.835639+0000","last_became_peered":"2024-01-21T04:33:47.835639+0000","last_unstale":"2024-01-21T05:05:24.986390+0000","last_undegraded":"2024-01-21T05:05:24.986390+0000","last_fullsized":"2024-01-21T05:05:24.986390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:55:33.768752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1048,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.953043+0000","last_change":"2024-01-21T04:35:26.902591+0000","last_active":"2024-01-21T05:05:26.953043+0000","last_peered":"2024-01-21T05:05:26.953043+0000","last_clean":"2024-01-21T05:05:26.953043+0000","last_became_active":"2024-01-21T04:35:26.889439+0000","last_became_peered":"2024-01-21T04:35:26.889439+0000","last_unstale":"2024-01-21T05:05:26.953043+0000","last_undegraded":"2024-01-21T05:05:26.953043+0000","last_fullsized":"2024-01-21T05:05:26.953043+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:36:16.061320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1054,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:43.860172+0000","last_change":"2024-01-21T04:33:48.112047+0000","last_active":"2024-01-21T05:05:43.860172+0000","last_peered":"2024-01-21T05:05:43.860172+0000","last_clean":"2024-01-21T05:05:43.860172+0000","last_became_active":"2024-01-21T04:33:48.080898+0000","last_became_peered":"2024-01-21T04:33:48.080898+0000","last_unstale":"2024-01-21T05:05:43.860172+0000","last_undegraded":"2024-01-21T05:05:43.860172+0000","last_fullsized":"2024-01-21T05:05:43.860172+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:54:29.756588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1064,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.082836+0000","last_change":"2024-01-21T04:35:36.906825+0000","last_active":"2024-01-21T05:05:25.082836+0000","last_peered":"2024-01-21T05:05:25.082836+0000","last_clean":"2024-01-21T05:05:25.082836+0000","last_became_active":"2024-01-21T04:35:36.901570+0000","last_became_peered":"2024-01-21T04:35:36.901570+0000","last_unstale":"2024-01-21T05:05:25.082836+0000","last_undegraded":"2024-01-21T05:05:25.082836+0000","last_fullsized":"2024-01-21T05:05:25.082836+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:17:42.698152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1055,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.015022+0000","last_change":"2024-01-21T04:33:48.051441+0000","last_active":"2024-01-21T05:05:25.015022+0000","last_peered":"2024-01-21T05:05:25.015022+0000","last_clean":"2024-01-21T05:05:25.015022+0000","last_became_active":"2024-01-21T04:33:48.046404+0000","last_became_peered":"2024-01-21T04:33:48.046404+0000","last_unstale":"2024-01-21T05:05:25.015022+0000","last_undegraded":"2024-01-21T05:05:25.015022+0000","last_fullsized":"2024-01-21T05:05:25.015022+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:19:24.446344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1048,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.039459+0000","last_change":"2024-01-21T04:35:19.621072+0000","last_active":"2024-01-21T05:05:26.039459+0000","last_peered":"2024-01-21T05:05:26.039459+0000","last_clean":"2024-01-21T05:05:26.039459+0000","last_became_active":"2024-01-21T04:35:19.611575+0000","last_became_peered":"2024-01-21T04:35:19.611575+0000","last_unstale":"2024-01-21T05:05:26.039459+0000","last_undegraded":"2024-01-21T05:05:26.039459+0000","last_fullsized":"2024-01-21T05:05:26.039459+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:09:33.751134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1082,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.249530+0000","last_change":"2024-01-21T04:34:53.506620+0000","last_active":"2024-01-21T05:05:25.249530+0000","last_peered":"2024-01-21T05:05:25.249530+0000","last_clean":"2024-01-21T05:05:25.249530+0000","last_became_active":"2024-01-21T04:34:53.495212+0000","last_became_peered":"2024-01-21T04:34:53.495212+0000","last_unstale":"2024-01-21T05:05:25.249530+0000","last_undegraded":"2024-01-21T05:05:25.249530+0000","last_fullsized":"2024-01-21T05:05:25.249530+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:38:49.385646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1046,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.949909+0000","last_change":"2024-01-21T04:34:58.579556+0000","last_active":"2024-01-21T05:05:26.949909+0000","last_peered":"2024-01-21T05:05:26.949909+0000","last_clean":"2024-01-21T05:05:26.949909+0000","last_became_active":"2024-01-21T04:34:58.492357+0000","last_became_peered":"2024-01-21T04:34:58.492357+0000","last_unstale":"2024-01-21T05:05:26.949909+0000","last_undegraded":"2024-01-21T05:05:26.949909+0000","last_fullsized":"2024-01-21T05:05:26.949909+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:55:43.736268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1046,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.954524+0000","last_change":"2024-01-21T04:35:26.240386+0000","last_active":"2024-01-21T05:05:26.954524+0000","last_peered":"2024-01-21T05:05:26.954524+0000","last_clean":"2024-01-21T05:05:26.954524+0000","last_became_active":"2024-01-21T04:35:25.928650+0000","last_became_peered":"2024-01-21T04:35:25.928650+0000","last_unstale":"2024-01-21T05:05:26.954524+0000","last_undegraded":"2024-01-21T05:05:26.954524+0000","last_fullsized":"2024-01-21T05:05:26.954524+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:38:53.872497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1062,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.970780+0000","last_change":"2024-01-21T04:35:00.395084+0000","last_active":"2024-01-21T05:05:24.970780+0000","last_peered":"2024-01-21T05:05:24.970780+0000","last_clean":"2024-01-21T05:05:24.970780+0000","last_became_active":"2024-01-21T04:35:00.387495+0000","last_became_peered":"2024-01-21T04:35:00.387495+0000","last_unstale":"2024-01-21T05:05:24.970780+0000","last_undegraded":"2024-01-21T05:05:24.970780+0000","last_fullsized":"2024-01-21T05:05:24.970780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:17:14.645461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1082,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.255370+0000","last_change":"2024-01-21T04:35:16.819308+0000","last_active":"2024-01-21T05:05:25.255370+0000","last_peered":"2024-01-21T05:05:25.255370+0000","last_clean":"2024-01-21T05:05:25.255370+0000","last_became_active":"2024-01-21T04:35:16.679256+0000","last_became_peered":"2024-01-21T04:35:16.679256+0000","last_unstale":"2024-01-21T05:05:25.255370+0000","last_undegraded":"2024-01-21T05:05:25.255370+0000","last_fullsized":"2024-01-21T05:05:25.255370+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:48:12.462617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1062,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:24.959859+0000","last_change":"2024-01-21T04:35:25.009419+0000","last_active":"2024-01-21T05:05:24.959859+0000","last_peered":"2024-01-21T05:05:24.959859+0000","last_clean":"2024-01-21T05:05:24.959859+0000","last_became_active":"2024-01-21T04:35:25.004049+0000","last_became_peered":"2024-01-21T04:35:25.004049+0000","last_unstale":"2024-01-21T05:05:24.959859+0000","last_undegraded":"2024-01-21T05:05:24.959859+0000","last_fullsized":"2024-01-21T05:05:24.959859+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:44:01.746389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1049,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.018026+0000","last_change":"2024-01-21T04:35:23.577107+0000","last_active":"2024-01-21T05:05:26.018026+0000","last_peered":"2024-01-21T05:05:26.018026+0000","last_clean":"2024-01-21T05:05:26.018026+0000","last_became_active":"2024-01-21T04:35:23.567833+0000","last_became_peered":"2024-01-21T04:35:23.567833+0000","last_unstale":"2024-01-21T05:05:26.018026+0000","last_undegraded":"2024-01-21T05:05:26.018026+0000","last_fullsized":"2024-01-21T05:05:26.018026+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:50:25.014755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1047,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.021801+0000","last_change":"2024-01-21T04:34:57.160123+0000","last_active":"2024-01-21T05:05:25.021801+0000","last_peered":"2024-01-21T05:05:25.021801+0000","last_clean":"2024-01-21T05:05:25.021801+0000","last_became_active":"2024-01-21T04:34:57.153977+0000","last_became_peered":"2024-01-21T04:34:57.153977+0000","last_unstale":"2024-01-21T05:05:25.021801+0000","last_undegraded":"2024-01-21T05:05:25.021801+0000","last_fullsized":"2024-01-21T05:05:25.021801+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:18:26.237354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1049,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:43.835339+0000","last_change":"2024-01-21T04:35:28.601417+0000","last_active":"2024-01-21T05:05:43.835339+0000","last_peered":"2024-01-21T05:05:43.835339+0000","last_clean":"2024-01-21T05:05:43.835339+0000","last_became_active":"2024-01-21T04:35:28.583632+0000","last_became_peered":"2024-01-21T04:35:28.583632+0000","last_unstale":"2024-01-21T05:05:43.835339+0000","last_undegraded":"2024-01-21T05:05:43.835339+0000","last_fullsized":"2024-01-21T05:05:43.835339+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:35:42.169701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1082,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.225971+0000","last_change":"2024-01-21T04:35:16.742449+0000","last_active":"2024-01-21T05:05:25.225971+0000","last_peered":"2024-01-21T05:05:25.225971+0000","last_clean":"2024-01-21T05:05:25.225971+0000","last_became_active":"2024-01-21T04:35:16.734793+0000","last_became_peered":"2024-01-21T04:35:16.734793+0000","last_unstale":"2024-01-21T05:05:25.225971+0000","last_undegraded":"2024-01-21T05:05:25.225971+0000","last_fullsized":"2024-01-21T05:05:25.225971+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:19:56.812663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1064,"reported_epoch":1106,"state":"active+clean","last_fresh":"2024-01-21T05:05:25.059570+0000","last_change":"2024-01-21T04:35:14.711183+0000","last_active":"2024-01-21T05:05:25.059570+0000","last_peered":"2024-01-21T05:05:25.059570+0000","last_clean":"2024-01-21T05:05:25.059570+0000","last_became_active":"2024-01-21T04:35:14.685816+0000","last_became_peered":"2024-01-21T04:35:14.685816+0000","last_unstale":"2024-01-21T05:05:25.059570+0000","last_undegraded":"2024-01-21T05:05:25.059570+0000","last_fullsized":"2024-01-21T05:05:25.059570+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:04:28.368206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":21,"seq":90194313604,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74468,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297372,"statfs":{"total":96636764160,"available":96560508928,"internally_reserved":0,"allocated":11628544,"data_stored":6358216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:02 2024","interfaces":[{"interface":"back","average":{"1min":16.902000000000001,"5min":11.343,"15min":17.483000000000001},"min":{"1min":6.3620000000000001,"5min":5.4320000000000004,"15min":4.5259999999999998},"max":{"1min":130.69499999999999,"5min":130.69499999999999,"15min":305.86900000000003},"last":17.004999999999999},{"interface":"front","average":{"1min":19.975999999999999,"5min":14.539,"15min":20.905000000000001},"min":{"1min":10.542999999999999,"5min":8.0370000000000008,"15min":4.6710000000000003},"max":{"1min":121.011,"5min":121.011,"15min":281.79000000000002},"last":19.651}]},{"osd":1,"last update":"Sun Jan 21 05:05:28 2024","interfaces":[{"interface":"back","average":{"1min":30.696999999999999,"5min":16.850999999999999,"15min":19.510000000000002},"min":{"1min":9.0579999999999998,"5min":7.6550000000000002,"15min":6.1280000000000001},"max":{"1min":151.977,"5min":151.977,"15min":249.82599999999999},"last":20.242999999999999},{"interface":"front","average":{"1min":21.411999999999999,"5min":10.917,"15min":14.314},"min":{"1min":5.5709999999999997,"5min":4.8860000000000001,"15min":4.4429999999999996},"max":{"1min":133.238,"5min":133.238,"15min":185.57599999999999},"last":12.459}]},{"osd":2,"last update":"Sun Jan 21 05:05:16 2024","interfaces":[{"interface":"back","average":{"1min":18.077000000000002,"5min":11.068,"15min":16.349},"min":{"1min":6.9770000000000003,"5min":6.4000000000000004,"15min":6.4000000000000004},"max":{"1min":102.258,"5min":102.258,"15min":260.14999999999998},"last":13.958},{"interface":"front","average":{"1min":24.271000000000001,"5min":15.379,"15min":22.163},"min":{"1min":9.6739999999999995,"5min":7.8390000000000004,"15min":7.8390000000000004},"max":{"1min":108.218,"5min":108.218,"15min":261.67200000000003},"last":15.444000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:05:07 2024","interfaces":[{"interface":"back","average":{"1min":21.986999999999998,"5min":12.903,"15min":16.385999999999999},"min":{"1min":6.5780000000000003,"5min":6.1299999999999999,"15min":6.0830000000000002},"max":{"1min":173.92099999999999,"5min":173.92099999999999,"15min":179.381},"last":14.923},{"interface":"front","average":{"1min":22.701000000000001,"5min":15.462,"15min":18.68},"min":{"1min":11.153,"5min":7.0839999999999996,"15min":6.766},"max":{"1min":120.02800000000001,"5min":120.02800000000001,"15min":209.67099999999999},"last":17.731999999999999}]},{"osd":5,"last update":"Sun Jan 21 05:04:58 2024","interfaces":[{"interface":"back","average":{"1min":15.048,"5min":15.337,"15min":22.681999999999999},"min":{"1min":11.866,"5min":7.8719999999999999,"15min":6.9800000000000004},"max":{"1min":17.161999999999999,"5min":30.702000000000002,"15min":318.983},"last":22.001000000000001},{"interface":"front","average":{"1min":12.109,"5min":12.140000000000001,"15min":21.193999999999999},"min":{"1min":8.077,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.620999999999999,"5min":21.742000000000001,"15min":341.21699999999998},"last":18.527000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:05:03 2024","interfaces":[{"interface":"back","average":{"1min":23.292000000000002,"5min":17.651,"15min":26.388999999999999},"min":{"1min":13.853999999999999,"5min":10.154999999999999,"15min":7.1859999999999999},"max":{"1min":110.64100000000001,"5min":110.64100000000001,"15min":329.39100000000002},"last":22.591000000000001},{"interface":"front","average":{"1min":24.414000000000001,"5min":19.693999999999999,"15min":27.457000000000001},"min":{"1min":13.991,"5min":11.544,"15min":11.026},"max":{"1min":102.995,"5min":102.995,"15min":330.54899999999998},"last":24.09}]},{"osd":7,"last update":"Sun Jan 21 05:05:28 2024","interfaces":[{"interface":"back","average":{"1min":40.505000000000003,"5min":19.132000000000001,"15min":21.704000000000001},"min":{"1min":9.1530000000000005,"5min":7.8639999999999999,"15min":7.7240000000000002},"max":{"1min":233.38499999999999,"5min":233.38499999999999,"15min":233.38499999999999},"last":20.902999999999999},{"interface":"front","average":{"1min":41.484999999999999,"5min":21.462,"15min":23.84},"min":{"1min":9.9809999999999999,"5min":9.9120000000000008,"15min":9.9120000000000008},"max":{"1min":264.34800000000001,"5min":264.34800000000001,"15min":264.34800000000001},"last":23.416}]}]},{"osd":6,"up_from":20,"seq":85899346308,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74476,"kb_used_data":11364,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297364,"statfs":{"total":96636764160,"available":96560500736,"internally_reserved":0,"allocated":11636736,"data_stored":6358225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":7.9119999999999999,"5min":8.2769999999999992,"15min":18.212},"min":{"1min":4.3540000000000001,"5min":4.109,"15min":4.0830000000000002},"max":{"1min":13.827,"5min":18.007999999999999,"15min":293.416},"last":5.2009999999999996},{"interface":"front","average":{"1min":8.2260000000000009,"5min":8.4949999999999992,"15min":17.937999999999999},"min":{"1min":4.3849999999999998,"5min":4.3849999999999998,"15min":4.2220000000000004},"max":{"1min":11.989000000000001,"5min":19.242000000000001,"15min":258.678},"last":6.7649999999999997}]},{"osd":1,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":12.247,"5min":11.308999999999999,"15min":17.893999999999998},"min":{"1min":7.6929999999999996,"5min":6.2839999999999998,"15min":6.0650000000000004},"max":{"1min":18.242999999999999,"5min":19.047999999999998,"15min":267.21899999999999},"last":12.834},{"interface":"front","average":{"1min":15.005000000000001,"5min":14.324,"15min":23.260000000000002},"min":{"1min":10.436999999999999,"5min":8.6660000000000004,"15min":7.3339999999999996},"max":{"1min":18.088999999999999,"5min":20.760000000000002,"15min":271.875},"last":13.657}]},{"osd":2,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":11.577,"5min":11.223000000000001,"15min":19.917999999999999},"min":{"1min":6.0039999999999996,"5min":6.0039999999999996,"15min":6.0039999999999996},"max":{"1min":19.081,"5min":19.081,"15min":274.54000000000002},"last":16.366},{"interface":"front","average":{"1min":14.459,"5min":13.849,"15min":17.952000000000002},"min":{"1min":7.5869999999999997,"5min":7.5869999999999997,"15min":5.9420000000000002},"max":{"1min":20.527000000000001,"5min":20.527000000000001,"15min":163.05199999999999},"last":18.417000000000002}]},{"osd":3,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":12.615,"5min":12.771000000000001,"15min":18.033999999999999},"min":{"1min":6.7919999999999998,"5min":6.7919999999999998,"15min":6.7549999999999999},"max":{"1min":18.863,"5min":20.734999999999999,"15min":156.47},"last":8.1890000000000001},{"interface":"front","average":{"1min":15.964,"5min":15.805,"15min":22.997},"min":{"1min":10.096,"5min":8.2469999999999999,"15min":8.2469999999999999},"max":{"1min":19.196999999999999,"5min":20.001999999999999,"15min":275.81},"last":15.608000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":17.013000000000002,"5min":16.920999999999999,"15min":25.783000000000001},"min":{"1min":13.029999999999999,"5min":11.298,"15min":9.625},"max":{"1min":20.202999999999999,"5min":21.173999999999999,"15min":280.101},"last":17.108000000000001},{"interface":"front","average":{"1min":18.748000000000001,"5min":18.567,"15min":22.736999999999998},"min":{"1min":15.997999999999999,"5min":11.999000000000001,"15min":6.4119999999999999},"max":{"1min":21.719000000000001,"5min":22.102,"15min":273.45600000000002},"last":17.837}]},{"osd":5,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":10.651999999999999,"5min":11.468,"15min":21.411999999999999},"min":{"1min":6.5499999999999998,"5min":6.1699999999999999,"15min":6.1699999999999999},"max":{"1min":20.835000000000001,"5min":20.835000000000001,"15min":275.233},"last":9.5609999999999999},{"interface":"front","average":{"1min":14.619,"5min":14.329000000000001,"15min":22.456},"min":{"1min":7.9710000000000001,"5min":7.9710000000000001,"15min":6.9500000000000002},"max":{"1min":23.047999999999998,"5min":23.047999999999998,"15min":288.76499999999999},"last":10.563000000000001}]},{"osd":7,"last update":"Sun Jan 21 05:04:52 2024","interfaces":[{"interface":"back","average":{"1min":12.044,"5min":12.678000000000001,"15min":21.946000000000002},"min":{"1min":8.0860000000000003,"5min":7.5220000000000002,"15min":7.5220000000000002},"max":{"1min":19.792000000000002,"5min":20.399000000000001,"15min":366.68000000000001},"last":11.395},{"interface":"front","average":{"1min":15.031000000000001,"5min":15.369,"15min":22.943999999999999},"min":{"1min":10.625,"5min":9.6790000000000003,"15min":9.6790000000000003},"max":{"1min":20.709,"5min":21.266999999999999,"15min":281.78500000000003},"last":15.943}]}]},{"osd":4,"up_from":20,"seq":85899346309,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74504,"kb_used_data":10904,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297336,"statfs":{"total":96636764160,"available":96560472064,"internally_reserved":0,"allocated":11165696,"data_stored":6053593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":17.173999999999999,"5min":11.048999999999999,"15min":16.907},"min":{"1min":5.7629999999999999,"5min":5.7629999999999999,"15min":4.2409999999999997},"max":{"1min":154.04499999999999,"5min":154.04499999999999,"15min":294.92099999999999},"last":6.4009999999999998},{"interface":"front","average":{"1min":18.936,"5min":14.742000000000001,"15min":19.247},"min":{"1min":8.673,"5min":6.4379999999999997,"15min":4.6699999999999999},"max":{"1min":99.890000000000001,"5min":99.890000000000001,"15min":292.43700000000001},"last":12.260999999999999}]},{"osd":1,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":10.364000000000001,"5min":9.9369999999999994,"15min":15.395},"min":{"1min":6.0350000000000001,"5min":5.1669999999999998,"15min":4.1470000000000002},"max":{"1min":60.579999999999998,"5min":60.579999999999998,"15min":221.84999999999999},"last":11.042},{"interface":"front","average":{"1min":12.541,"5min":13.403,"15min":17.917000000000002},"min":{"1min":7.2190000000000003,"5min":7.032,"15min":6.3959999999999999},"max":{"1min":22.050000000000001,"5min":36.454999999999998,"15min":299.11599999999999},"last":20.722000000000001}]},{"osd":2,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":13.195,"5min":13.805999999999999,"15min":19.687999999999999},"min":{"1min":7.6289999999999996,"5min":7.0209999999999999,"15min":5.7439999999999998},"max":{"1min":23.762,"5min":30.693999999999999,"15min":293.60199999999998},"last":14.542},{"interface":"front","average":{"1min":13.038,"5min":11.211,"15min":20.07},"min":{"1min":7.532,"5min":6.4829999999999997,"15min":6.4829999999999997},"max":{"1min":20.181999999999999,"5min":34.270000000000003,"15min":303.94799999999998},"last":15.597}]},{"osd":3,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":29.370000000000001,"5min":16.681000000000001,"15min":22.192},"min":{"1min":6.0940000000000003,"5min":6.0940000000000003,"15min":6.0940000000000003},"max":{"1min":226.172,"5min":226.172,"15min":340.21100000000001},"last":18.847999999999999},{"interface":"front","average":{"1min":36.143999999999998,"5min":16.957999999999998,"15min":22.98},"min":{"1min":9.0310000000000006,"5min":6.6529999999999996,"15min":6.5579999999999998},"max":{"1min":211.38300000000001,"5min":211.38300000000001,"15min":302.72500000000002},"last":10.221}]},{"osd":5,"last update":"Sun Jan 21 05:04:58 2024","interfaces":[{"interface":"back","average":{"1min":23.199999999999999,"5min":15.675000000000001,"15min":23.965},"min":{"1min":8.4740000000000002,"5min":7.2510000000000003,"15min":7.2510000000000003},"max":{"1min":206.16800000000001,"5min":206.16800000000001,"15min":305.17200000000003},"last":21.571999999999999},{"interface":"front","average":{"1min":24.972999999999999,"5min":18.754999999999999,"15min":29.27},"min":{"1min":9.9359999999999999,"5min":9.9359999999999999,"15min":9.9359999999999999},"max":{"1min":192.066,"5min":192.066,"15min":326.19900000000001},"last":22.553999999999998}]},{"osd":6,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":24.899999999999999,"5min":15.957000000000001,"15min":23.010999999999999},"min":{"1min":6.8769999999999998,"5min":6.7409999999999997,"15min":6.7409999999999997},"max":{"1min":204.32900000000001,"5min":204.32900000000001,"15min":248.09800000000001},"last":8.3130000000000006},{"interface":"front","average":{"1min":27.047000000000001,"5min":18.641999999999999,"15min":26.821000000000002},"min":{"1min":7.7800000000000002,"5min":7.7800000000000002,"15min":7.7800000000000002},"max":{"1min":205.94800000000001,"5min":205.94800000000001,"15min":333.46899999999999},"last":16.577999999999999}]},{"osd":7,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":28.289999999999999,"5min":17.347000000000001,"15min":20.901},"min":{"1min":9.5459999999999994,"5min":7.3970000000000002,"15min":6.5430000000000001},"max":{"1min":260.63499999999999,"5min":260.63499999999999,"15min":312.41199999999998},"last":17.927},{"interface":"front","average":{"1min":29.556000000000001,"5min":19.172999999999998,"15min":25.035},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":236.80799999999999,"5min":236.80799999999999,"15min":247.44499999999999},"last":13.315}]}]},{"osd":7,"up_from":15,"seq":64424509829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74500,"kb_used_data":11144,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297340,"statfs":{"total":96636764160,"available":96560476160,"internally_reserved":0,"allocated":11411456,"data_stored":6213897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:21 2024","interfaces":[{"interface":"back","average":{"1min":26.605,"5min":13.619999999999999,"15min":20.289000000000001},"min":{"1min":4.9279999999999999,"5min":4.46,"15min":3.847},"max":{"1min":244.893,"5min":244.893,"15min":272.279},"last":8.2669999999999995},{"interface":"front","average":{"1min":27.978000000000002,"5min":12.045999999999999,"15min":19.513000000000002},"min":{"1min":4.516,"5min":4.0579999999999998,"15min":4.0579999999999998},"max":{"1min":224.22800000000001,"5min":224.22800000000001,"15min":356.99799999999999},"last":6.202}]},{"osd":1,"last update":"Sun Jan 21 05:04:54 2024","interfaces":[{"interface":"back","average":{"1min":8.9190000000000005,"5min":8.7639999999999993,"15min":15.289999999999999},"min":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7759999999999998},"max":{"1min":19.866,"5min":19.866,"15min":276.94099999999997},"last":9.0719999999999992},{"interface":"front","average":{"1min":13.728,"5min":13.041,"15min":20.687999999999999},"min":{"1min":7.8600000000000003,"5min":7.4729999999999999,"15min":6.383},"max":{"1min":23.332000000000001,"5min":23.332000000000001,"15min":232.74299999999999},"last":9.6750000000000007}]},{"osd":2,"last update":"Sun Jan 21 05:05:21 2024","interfaces":[{"interface":"back","average":{"1min":32.673999999999999,"5min":17.591000000000001,"15min":25.193000000000001},"min":{"1min":8.125,"5min":7.5919999999999996,"15min":7.5919999999999996},"max":{"1min":259.29899999999998,"5min":259.29899999999998,"15min":284.34100000000001},"last":10.281000000000001},{"interface":"front","average":{"1min":35.651000000000003,"5min":20.353000000000002,"15min":24.027999999999999},"min":{"1min":14.430999999999999,"5min":11.855,"15min":6.0229999999999997},"max":{"1min":260.87799999999999,"5min":260.87799999999999,"15min":260.87799999999999},"last":16.260000000000002}]},{"osd":3,"last update":"Sun Jan 21 05:05:37 2024","interfaces":[{"interface":"back","average":{"1min":41.347000000000001,"5min":21.199000000000002,"15min":26.056000000000001},"min":{"1min":8.8219999999999992,"5min":8.7919999999999998,"15min":7.8109999999999999},"max":{"1min":263.053,"5min":263.053,"15min":276.61399999999998},"last":11.595000000000001},{"interface":"front","average":{"1min":44.539999999999999,"5min":23.657,"15min":29.605},"min":{"1min":14.435,"5min":13.215,"15min":10.497},"max":{"1min":265.154,"5min":265.154,"15min":277.28100000000001},"last":16.835000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:05:37 2024","interfaces":[{"interface":"back","average":{"1min":50.813000000000002,"5min":18.835000000000001,"15min":17.899000000000001},"min":{"1min":7.5730000000000004,"5min":5.8620000000000001,"15min":5.8620000000000001},"max":{"1min":316.40699999999998,"5min":316.40699999999998,"15min":316.40699999999998},"last":10.880000000000001},{"interface":"front","average":{"1min":45.965000000000003,"5min":23.863,"15min":25.629000000000001},"min":{"1min":15.303000000000001,"5min":12.269,"15min":9.1609999999999996},"max":{"1min":256.67099999999999,"5min":256.67099999999999,"15min":263.23099999999999},"last":18.271999999999998}]},{"osd":5,"last update":"Sun Jan 21 05:05:22 2024","interfaces":[{"interface":"back","average":{"1min":38.140999999999998,"5min":16.295000000000002,"15min":21.081},"min":{"1min":8.4489999999999998,"5min":6.6459999999999999,"15min":6.6429999999999998},"max":{"1min":223.56999999999999,"5min":223.56999999999999,"15min":223.56999999999999},"last":12.335000000000001},{"interface":"front","average":{"1min":40.212000000000003,"5min":19.149000000000001,"15min":23.727},"min":{"1min":8.6560000000000006,"5min":8.6240000000000006,"15min":8.5429999999999993},"max":{"1min":224.81100000000001,"5min":224.81100000000001,"15min":224.81100000000001},"last":14.775}]},{"osd":6,"last update":"Sun Jan 21 05:04:54 2024","interfaces":[{"interface":"back","average":{"1min":13.615,"5min":13.750999999999999,"15min":26.07},"min":{"1min":8.2759999999999998,"5min":7.2210000000000001,"15min":7.2210000000000001},"max":{"1min":31.048999999999999,"5min":31.048999999999999,"15min":371.17599999999999},"last":14.109},{"interface":"front","average":{"1min":13.039999999999999,"5min":13.173,"15min":19.331},"min":{"1min":7.931,"5min":7.5679999999999996,"15min":7.1020000000000003},"max":{"1min":22.701000000000001,"5min":22.701000000000001,"15min":179.63900000000001},"last":15.513999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509825,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78648,"kb_used_data":11132,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94293192,"statfs":{"total":96636764160,"available":96556228608,"internally_reserved":0,"allocated":11399168,"data_stored":6207364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":9.6099999999999994,"5min":9.1780000000000008,"15min":20.884},"min":{"1min":5.1840000000000002,"5min":4.7140000000000004,"15min":4.5629999999999997},"max":{"1min":14.326000000000001,"5min":18.608000000000001,"15min":273.28300000000002},"last":21.396999999999998},{"interface":"front","average":{"1min":8.9429999999999996,"5min":9.4139999999999997,"15min":19.279},"min":{"1min":5.1059999999999999,"5min":4.9880000000000004,"15min":4.9880000000000004},"max":{"1min":16.013999999999999,"5min":18.042000000000002,"15min":278.84500000000003},"last":19.457999999999998}]},{"osd":2,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":11.167999999999999,"5min":10.718999999999999,"15min":22.236999999999998},"min":{"1min":6.7869999999999999,"5min":5.5469999999999997,"15min":5.5469999999999997},"max":{"1min":17.061,"5min":18.632999999999999,"15min":404.36700000000002},"last":15.826000000000001},{"interface":"front","average":{"1min":15.44,"5min":15.366,"15min":29.544},"min":{"1min":10.481999999999999,"5min":9.4169999999999998,"15min":8.8650000000000002},"max":{"1min":21.687999999999999,"5min":22.27,"15min":407.68000000000001},"last":17.228000000000002}]},{"osd":3,"last update":"Sun Jan 21 05:05:06 2024","interfaces":[{"interface":"back","average":{"1min":26.777000000000001,"5min":17.292000000000002,"15min":25.620999999999999},"min":{"1min":9.8919999999999995,"5min":6.266,"15min":6.266},"max":{"1min":212.34200000000001,"5min":212.34200000000001,"15min":253.25399999999999},"last":20.538},{"interface":"front","average":{"1min":13.295999999999999,"5min":10.103999999999999,"15min":17.751999999999999},"min":{"1min":5.7930000000000001,"5min":5.7930000000000001,"15min":5.7930000000000001},"max":{"1min":63.148000000000003,"5min":63.148000000000003,"15min":315.92399999999998},"last":7.5389999999999997}]},{"osd":4,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":10.625999999999999,"5min":11.837,"15min":20.167999999999999},"min":{"1min":7.5839999999999996,"5min":7.0149999999999997,"15min":6.2089999999999996},"max":{"1min":21.039999999999999,"5min":22.332999999999998,"15min":283.37599999999998},"last":10.917},{"interface":"front","average":{"1min":16.094999999999999,"5min":16.707000000000001,"15min":28.533000000000001},"min":{"1min":11.058999999999999,"5min":9.7360000000000007,"15min":9.7360000000000007},"max":{"1min":21.434999999999999,"5min":23.916,"15min":344.37200000000001},"last":22.501999999999999}]},{"osd":5,"last update":"Sun Jan 21 05:04:42 2024","interfaces":[{"interface":"back","average":{"1min":13.879,"5min":14.018000000000001,"15min":26.966000000000001},"min":{"1min":7.4290000000000003,"5min":7.4290000000000003,"15min":6.6029999999999998},"max":{"1min":18.428999999999998,"5min":20.844000000000001,"15min":354.04300000000001},"last":16.457999999999998},{"interface":"front","average":{"1min":12.494999999999999,"5min":12.369999999999999,"15min":23.998000000000001},"min":{"1min":8.6600000000000001,"5min":7.7089999999999996,"15min":6.3159999999999998},"max":{"1min":17.739999999999998,"5min":22.294,"15min":296.52499999999998},"last":10.146000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:05:02 2024","interfaces":[{"interface":"back","average":{"1min":25.177,"5min":16.742000000000001,"15min":24.870000000000001},"min":{"1min":10.148,"5min":8.1799999999999997,"15min":6.5659999999999998},"max":{"1min":226.14599999999999,"5min":226.14599999999999,"15min":288.29599999999999},"last":12.824},{"interface":"front","average":{"1min":27.905999999999999,"5min":19.323,"15min":29.056000000000001},"min":{"1min":11.382,"5min":10.058,"15min":8.8550000000000004},"max":{"1min":236.083,"5min":236.083,"15min":293.56400000000002},"last":18.486000000000001}]},{"osd":7,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":28.417999999999999,"5min":15.923999999999999,"15min":23.308},"min":{"1min":9.2400000000000002,"5min":8.0700000000000003,"15min":7.6509999999999998},"max":{"1min":113.319,"5min":113.319,"15min":329.11200000000002},"last":11.997999999999999},{"interface":"front","average":{"1min":33.444000000000003,"5min":19.013000000000002,"15min":28.425999999999998},"min":{"1min":10.021000000000001,"5min":10.021000000000001,"15min":9.3699999999999992},"max":{"1min":115.97,"5min":115.97,"15min":337.24000000000001},"last":13.478999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509827,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82788,"kb_used_data":11356,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289052,"statfs":{"total":96636764160,"available":96551989248,"internally_reserved":0,"allocated":11628544,"data_stored":6358216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":14.563000000000001,"5min":11.273999999999999,"15min":15.429},"min":{"1min":4.4800000000000004,"5min":4.4800000000000004,"15min":4.343},"max":{"1min":68.804000000000002,"5min":68.804000000000002,"15min":188.334},"last":7.4740000000000002},{"interface":"front","average":{"1min":10.598000000000001,"5min":9.1699999999999999,"15min":14.536},"min":{"1min":5.8540000000000001,"5min":5.0380000000000003,"15min":4.2919999999999998},"max":{"1min":26.100000000000001,"5min":26.100000000000001,"15min":153.12299999999999},"last":9.3000000000000007}]},{"osd":1,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":13.564,"5min":10.316000000000001,"15min":16.317},"min":{"1min":5.6440000000000001,"5min":5.5419999999999998,"15min":5.5419999999999998},"max":{"1min":37.771999999999998,"5min":37.771999999999998,"15min":318.404},"last":11.583},{"interface":"front","average":{"1min":19.129000000000001,"5min":14.199,"15min":21.916},"min":{"1min":10.805999999999999,"5min":7.1699999999999999,"15min":7.1699999999999999},"max":{"1min":58.195,"5min":58.195,"15min":333.91000000000003},"last":15.878}]},{"osd":3,"last update":"Sun Jan 21 05:05:24 2024","interfaces":[{"interface":"back","average":{"1min":17.065000000000001,"5min":12.718,"15min":18.966999999999999},"min":{"1min":7.1639999999999997,"5min":6.9340000000000002,"15min":5.7240000000000002},"max":{"1min":48.807000000000002,"5min":48.807000000000002,"15min":314.58600000000001},"last":12.199999999999999},{"interface":"front","average":{"1min":19.007000000000001,"5min":15.404999999999999,"15min":22.751000000000001},"min":{"1min":9.9469999999999992,"5min":9.7530000000000001,"15min":8.5250000000000004},"max":{"1min":71.736999999999995,"5min":71.736999999999995,"15min":335.71699999999998},"last":10.006}]},{"osd":4,"last update":"Sun Jan 21 05:04:39 2024","interfaces":[{"interface":"back","average":{"1min":12.945,"5min":13.516,"15min":19.908000000000001},"min":{"1min":8.6189999999999998,"5min":8.25,"15min":6.0999999999999996},"max":{"1min":18.849,"5min":23.408000000000001,"15min":313.17200000000003},"last":16.82},{"interface":"front","average":{"1min":15.632999999999999,"5min":15.916,"15min":22.266999999999999},"min":{"1min":10.994,"5min":9.593,"15min":6.1920000000000002},"max":{"1min":18.355,"5min":27.402999999999999,"15min":331.18200000000002},"last":14.096}]},{"osd":5,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":20.053999999999998,"5min":12.343999999999999,"15min":19.91},"min":{"1min":6.8529999999999998,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":166.327,"5min":166.327,"15min":270.91399999999999},"last":8.4440000000000008},{"interface":"front","average":{"1min":24.475999999999999,"5min":17.946000000000002,"15min":24.292999999999999},"min":{"1min":13.491,"5min":10.361000000000001,"15min":8.8279999999999994},"max":{"1min":130.548,"5min":130.548,"15min":259.60399999999998},"last":17.736000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:04:51 2024","interfaces":[{"interface":"back","average":{"1min":17.291,"5min":17.452999999999999,"15min":26.175000000000001},"min":{"1min":12.855,"5min":11.922000000000001,"15min":7.9800000000000004},"max":{"1min":20.937000000000001,"5min":31.047000000000001,"15min":321.79500000000002},"last":19.460000000000001},{"interface":"front","average":{"1min":12.227,"5min":12.388,"15min":21.454999999999998},"min":{"1min":8.5169999999999995,"5min":7.9699999999999998,"15min":7.7229999999999999},"max":{"1min":20.093,"5min":28.638999999999999,"15min":282.54599999999999},"last":12.933}]},{"osd":7,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":24.565000000000001,"5min":19.756,"15min":26.001000000000001},"min":{"1min":16.887,"5min":13.028,"15min":8.8460000000000001},"max":{"1min":67.106999999999999,"5min":67.106999999999999,"15min":312.435},"last":18.59},{"interface":"front","average":{"1min":20.120000000000001,"5min":15.542,"15min":23.835000000000001},"min":{"1min":8.5540000000000003,"5min":8.2370000000000001,"15min":8.2370000000000001},"max":{"1min":97.379000000000005,"5min":97.379000000000005,"15min":256.11900000000003},"last":14.731999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509825,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74864,"kb_used_data":11752,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94296976,"statfs":{"total":96636764160,"available":96560103424,"internally_reserved":0,"allocated":12034048,"data_stored":6756061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:04:45 2024","interfaces":[{"interface":"back","average":{"1min":8.1910000000000007,"5min":8.7469999999999999,"15min":16.152999999999999},"min":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"max":{"1min":17.446000000000002,"5min":27.192,"15min":209.96299999999999},"last":7.2469999999999999},{"interface":"front","average":{"1min":9.2430000000000003,"5min":9.1839999999999993,"15min":17.379000000000001},"min":{"1min":5.3650000000000002,"5min":4.9560000000000004,"15min":4.7549999999999999},"max":{"1min":14.426,"5min":25.323,"15min":244.054},"last":8.2629999999999999}]},{"osd":2,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":25.413,"5min":13.148999999999999,"15min":17.792999999999999},"min":{"1min":8.1020000000000003,"5min":6.202,"15min":4.8819999999999997},"max":{"1min":218.80500000000001,"5min":218.80500000000001,"15min":221.54300000000001},"last":8.0009999999999994},{"interface":"front","average":{"1min":24.704999999999998,"5min":15.188000000000001,"15min":20.256},"min":{"1min":10.317,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":144.12899999999999,"5min":144.12899999999999,"15min":219.34899999999999},"last":13.866}]},{"osd":3,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":14.996,"5min":13.903,"15min":21.224},"min":{"1min":7.3200000000000003,"5min":6.569,"15min":6.569},"max":{"1min":40.685000000000002,"5min":40.685000000000002,"15min":163.58799999999999},"last":17.402000000000001},{"interface":"front","average":{"1min":14.272,"5min":12.506,"15min":19.687999999999999},"min":{"1min":7.9630000000000001,"5min":7.0549999999999997,"15min":6.4809999999999999},"max":{"1min":55.024999999999999,"5min":55.024999999999999,"15min":244.178},"last":15.630000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:04:45 2024","interfaces":[{"interface":"back","average":{"1min":15.436999999999999,"5min":15.552,"15min":21.210000000000001},"min":{"1min":7.6369999999999996,"5min":7.6369999999999996,"15min":7.6369999999999996},"max":{"1min":19.053000000000001,"5min":23.524999999999999,"15min":241.94800000000001},"last":18.265000000000001},{"interface":"front","average":{"1min":10.997,"5min":10.789999999999999,"15min":18.545000000000002},"min":{"1min":7.0789999999999997,"5min":6.7069999999999999,"15min":6.7069999999999999},"max":{"1min":18.260999999999999,"5min":18.260999999999999,"15min":225.60499999999999},"last":12.446999999999999}]},{"osd":5,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":26.631,"5min":16.741,"15min":22.994},"min":{"1min":9.1850000000000005,"5min":9.1850000000000005,"15min":6.8970000000000002},"max":{"1min":156.554,"5min":156.554,"15min":313.36599999999999},"last":10.292999999999999},{"interface":"front","average":{"1min":26.550000000000001,"5min":16.300999999999998,"15min":23.606000000000002},"min":{"1min":8.9369999999999994,"5min":8.9369999999999994,"15min":5.851},"max":{"1min":138.114,"5min":138.114,"15min":298.49599999999998},"last":9.0289999999999999}]},{"osd":6,"last update":"Sun Jan 21 05:04:45 2024","interfaces":[{"interface":"back","average":{"1min":15.303000000000001,"5min":15.319000000000001,"15min":24.856000000000002},"min":{"1min":9.827,"5min":8.8949999999999996,"15min":8.8949999999999996},"max":{"1min":19.283000000000001,"5min":23.902999999999999,"15min":298.51999999999998},"last":11.117000000000001},{"interface":"front","average":{"1min":16.611999999999998,"5min":16.690999999999999,"15min":24.472999999999999},"min":{"1min":11.101000000000001,"5min":10.550000000000001,"15min":7.4880000000000004},"max":{"1min":21.846,"5min":26.234000000000002,"15min":181.01599999999999},"last":13.145}]},{"osd":7,"last update":"Sun Jan 21 05:05:17 2024","interfaces":[{"interface":"back","average":{"1min":21.521000000000001,"5min":16.407,"15min":24.297999999999998},"min":{"1min":11.332000000000001,"5min":9.8070000000000004,"15min":9.3230000000000004},"max":{"1min":107.875,"5min":107.875,"15min":214.24199999999999},"last":11.731},{"interface":"front","average":{"1min":20.009,"5min":17.219999999999999,"15min":24.443000000000001},"min":{"1min":9.5779999999999994,"5min":9.5779999999999994,"15min":9.4120000000000008},"max":{"1min":51.546999999999997,"5min":51.546999999999997,"15min":180.73099999999999},"last":16.562999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575233,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74860,"kb_used_data":11748,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94296980,"statfs":{"total":96636764160,"available":96560107520,"internally_reserved":0,"allocated":12029952,"data_stored":6756056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":23.058,"5min":12.195,"15min":18.460000000000001},"min":{"1min":6.226,"5min":5.2300000000000004,"15min":4.601},"max":{"1min":154.13399999999999,"5min":154.13399999999999,"15min":377.50999999999999},"last":13.148},{"interface":"front","average":{"1min":32.405999999999999,"5min":17.396999999999998,"15min":21.706},"min":{"1min":7.8029999999999999,"5min":5.8559999999999999,"15min":4.4359999999999999},"max":{"1min":193.01400000000001,"5min":193.01400000000001,"15min":224.58600000000001},"last":22.873999999999999}]},{"osd":1,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":24.754999999999999,"5min":14.738,"15min":22.327999999999999},"min":{"1min":6.165,"5min":5.6820000000000004,"15min":4.5380000000000003},"max":{"1min":160.279,"5min":160.279,"15min":368.79199999999997},"last":10.007},{"interface":"front","average":{"1min":28.187000000000001,"5min":13.547000000000001,"15min":19.158999999999999},"min":{"1min":9.9169999999999998,"5min":4.944,"15min":4.3170000000000002},"max":{"1min":156.38499999999999,"5min":156.38499999999999,"15min":225.21000000000001},"last":7.5839999999999996}]},{"osd":2,"last update":"Sun Jan 21 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":28.533000000000001,"5min":14.004,"15min":20.603999999999999},"min":{"1min":7.7069999999999999,"5min":6.6420000000000003,"15min":5.8460000000000001},"max":{"1min":232.78200000000001,"5min":232.78200000000001,"15min":232.78200000000001},"last":11.324999999999999},{"interface":"front","average":{"1min":35.103000000000002,"5min":15.885999999999999,"15min":22.988},"min":{"1min":7.9740000000000002,"5min":6.4199999999999999,"15min":5.8289999999999997},"max":{"1min":288.49099999999999,"5min":288.49099999999999,"15min":309.24599999999998},"last":6.6349999999999998}]},{"osd":3,"last update":"Sun Jan 21 05:04:44 2024","interfaces":[{"interface":"back","average":{"1min":15.166,"5min":15.417,"15min":25.603000000000002},"min":{"1min":11.159000000000001,"5min":8.516,"15min":6.7229999999999999},"max":{"1min":18.838000000000001,"5min":31.256,"15min":375.07799999999997},"last":12.566000000000001},{"interface":"front","average":{"1min":12.089,"5min":11.997999999999999,"15min":22.629999999999999},"min":{"1min":7.6299999999999999,"5min":6.4589999999999996,"15min":6.4589999999999996},"max":{"1min":16.795999999999999,"5min":32.795999999999999,"15min":276.57100000000003},"last":14.372}]},{"osd":4,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":31.202999999999999,"5min":15.868,"15min":24.077000000000002},"min":{"1min":8.9039999999999999,"5min":6.3360000000000003,"15min":6.3360000000000003},"max":{"1min":184.767,"5min":184.767,"15min":272.274},"last":8.3659999999999997},{"interface":"front","average":{"1min":33.640999999999998,"5min":20.071999999999999,"15min":28.475000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":6.8689999999999998},"max":{"1min":204.47499999999999,"5min":204.47499999999999,"15min":378.404},"last":21.052}]},{"osd":6,"last update":"Sun Jan 21 05:04:44 2024","interfaces":[{"interface":"back","average":{"1min":13.416,"5min":12.768000000000001,"15min":23.533999999999999},"min":{"1min":9.0690000000000008,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":19.172000000000001,"5min":32.139000000000003,"15min":296.88499999999999},"last":17.539999999999999},{"interface":"front","average":{"1min":16.545999999999999,"5min":16.481999999999999,"15min":30.184999999999999},"min":{"1min":11.481999999999999,"5min":10.188000000000001,"15min":9.6750000000000007},"max":{"1min":20.914000000000001,"5min":37.082999999999998,"15min":311.10899999999998},"last":18.202999999999999}]},{"osd":7,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":31.890000000000001,"5min":20.614999999999998,"15min":29.431999999999999},"min":{"1min":10.763,"5min":9.9640000000000004,"15min":8.032},"max":{"1min":202.90700000000001,"5min":202.90700000000001,"15min":486.358},"last":21.878},{"interface":"front","average":{"1min":36.167000000000002,"5min":20.385999999999999,"15min":27.478999999999999},"min":{"1min":10.587999999999999,"5min":9.8740000000000006,"15min":9.1039999999999992},"max":{"1min":177.25800000000001,"5min":177.25800000000001,"15min":486.62900000000002},"last":18.789000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T05:05:46.367 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.465+0000 7f4bac4e5640 10 monclient: get_monmap_and_config 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.465+0000 7f4bac4e5640 10 monclient: build_initial_monmap 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.465+0000 7f4bac4e5640 10 monclient: monmap: 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:46.466229+0000 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:46.466229+0000 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:46.466 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:46.467 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:46.467 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:46.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.465+0000 7f4bac4e5640 1 Processor -- start 2024-01-21T05:05:46.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.465+0000 7f4bac4e5640 1 -- start start 2024-01-21T05:05:46.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.465+0000 7f4bac4e5640 10 monclient: init 2024-01-21T05:05:46.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ba40a8a70 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ba40a8e50 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f4ba413f6d0 0x7f4ba414d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 10 monclient(hunting): picked mon.noname-c con 0x7f4ba413f6d0 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba40aa690 con 0x7f4ba40a8a70 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba40aa970 con 0x7f4ba40a8e50 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba40a9f90 con 0x7f4ba413f6d0 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.466+0000 7f4bac4e5640 10 monclient(hunting): authenticate will time out at 3304.807782s 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.467+0000 7f4baa281640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4ba40a8a70 0x7f4ba413efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50192/0 (socket says 172.21.15.69:50192) 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.467+0000 7f4baa281640 1 -- 172.21.15.69:0/3231231247 learned_addr learned my addr 172.21.15.69:0/3231231247 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.467+0000 7f4baaa82640 1 --2- 172.21.15.69:0/3231231247 >> v2:172.21.15.144:3300/0 conn(0x7f4ba413f6d0 0x7f4ba414d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.467+0000 7f4baaa82640 10 monclient(hunting): get_auth_request con 0x7f4ba413f6d0 auth_method 0 2024-01-21T05:05:46.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.467+0000 7f4baaa82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:46.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.467+0000 7f4baaa82640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:46.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.467+0000 7f4baaa82640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:46.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.468+0000 7f4baaa82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:46.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.468+0000 7f4baaa82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:46.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.468+0000 7f4baaa82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4ba927f640 1 -- 172.21.15.69:0/3231231247 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310376685 0 0) 0x7f4ba40aa690 con 0x7f4ba40a8a70 2024-01-21T05:05:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4ba927f640 10 monclient(hunting): _negotiate 2024-01-21T05:05:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4ba927f640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4ba927f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4ba927f640 10 monclient(hunting): my global_id is 17430 2024-01-21T05:05:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4ba927f640 1 -- 172.21.15.69:0/3231231247 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b90003520 con 0x7f4ba40a8a70 2024-01-21T05:05:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4baaa82640 10 monclient(hunting): handle_auth_done global_id 21541 payload 274 2024-01-21T05:05:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4baaa82640 1 -- 172.21.15.69:0/3231231247 >> v1:172.21.15.144:6789/0 conn(0x7f4ba40a8e50 legacy=0x7f4ba4142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4baaa82640 1 -- 172.21.15.69:0/3231231247 >> v1:172.21.15.69:6789/0 conn(0x7f4ba40a8a70 legacy=0x7f4ba413efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4baaa82640 10 monclient: _finish_hunting 0 2024-01-21T05:05:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4baaa82640 1 monclient: found mon.noname-c 2024-01-21T05:05:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4baaa82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4baaa82640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T05:05:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.469+0000 7f4baaa82640 1 -- 172.21.15.69:0/3231231247 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba40aac50 con 0x7f4ba413f6d0 2024-01-21T05:05:46.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.473+0000 7f4baaa82640 1 --2- 172.21.15.69:0/3231231247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ba413f6d0 0x7f4ba414d8b0 secure :-1 s=READY pgs=5217 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00097a0 tx=0x7f4ba002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=802208aa7447ce1d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 1 -- 172.21.15.69:0/3231231247 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba002e070 con 0x7f4ba413f6d0 2024-01-21T05:05:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:05:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 10 monclient: dump: 2024-01-21T05:05:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 10 monclient: _finish_auth 0 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 10 monclient: _check_auth_tickets 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 1 -- 172.21.15.69:0/3231231247 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba00315a0 con 0x7f4ba413f6d0 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4bac4e5640 5 monclient: authenticate success, global_id 21541 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4bac4e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4bac4e5640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 1 -- 172.21.15.69:0/3231231247 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0031930 con 0x7f4ba413f6d0 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4bac4e5640 4 set_mon_vals no callback set 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.476+0000 7f4ba927f640 10 monclient: dump: 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:46.480 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.477+0000 7f4bac4e5640 10 monclient: shutdown 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.477+0000 7f4bac4e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.477+0000 7f4bac4e5640 1 -- 172.21.15.69:0/3231231247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ba413f6d0 msgr2=0x7f4ba414d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.477+0000 7f4bac4e5640 1 --2- 172.21.15.69:0/3231231247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ba413f6d0 0x7f4ba414d8b0 secure :-1 s=READY pgs=5217 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00097a0 tx=0x7f4ba002b840 comp rx=0 tx=0).stop 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.477+0000 7f4bac4e5640 1 -- 172.21.15.69:0/3231231247 shutdown_connections 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.477+0000 7f4bac4e5640 1 --2- 172.21.15.69:0/3231231247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ba413f6d0 0x7f4ba414d8b0 unknown :-1 s=CLOSED pgs=5217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.477+0000 7f4bac4e5640 1 -- 172.21.15.69:0/3231231247 >> 172.21.15.69:0/3231231247 conn(0x7f4ba4066f00 msgr2=0x7f4ba40b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.477+0000 7f4bac4e5640 1 -- 172.21.15.69:0/3231231247 shutdown_connections 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.477+0000 7f4bac4e5640 1 -- 172.21.15.69:0/3231231247 wait complete. 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.478+0000 7f4bac4e5640 10 monclient: build_initial_monmap 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.478+0000 7f4bac4e5640 10 monclient: monmap: 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.478+0000 7f4bac4e5640 1 librados: starting msgr at 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.478+0000 7f4bac4e5640 1 librados: starting objecter 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.478+0000 7f4bac4e5640 1 Processor -- start 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.478+0000 7f4bac4e5640 1 -- start start 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.478+0000 7f4bac4e5640 1 librados: setting wanted keys 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.478+0000 7f4bac4e5640 1 librados: calling monclient init 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.478+0000 7f4bac4e5640 10 monclient: init 2024-01-21T05:05:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ba40a8a70 0x7f4ba40aba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 10 monclient(hunting): picked mon.noname-c con 0x7f4ba40a8a70 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ba40a8e50 0x7f4ba40ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 10 monclient(hunting): picked mon.noname-a con 0x7f4ba40a8e50 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ba413f6d0 0x7f4ba41d7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 10 monclient(hunting): picked mon.noname-b con 0x7f4ba413f6d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba41d6df0 con 0x7f4ba40a8e50 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba40ade60 con 0x7f4ba413f6d0 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba4144360 con 0x7f4ba40a8a70 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4bac4e5640 10 monclient(hunting): authenticate will time out at 3304.820457s 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4baa281640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ba40a8a70 0x7f4ba40aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.479+0000 7f4ba9a80640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ba40a8e50 0x7f4ba40ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baaa82640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ba413f6d0 0x7f4ba41d7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4ba9a80640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ba40a8e50 0x7f4ba40ac440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41574/0 (socket says 172.21.15.69:41574) 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4ba9a80640 1 -- 172.21.15.69:0/2596904798 learned_addr learned my addr 172.21.15.69:0/2596904798 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4ba9a80640 10 monclient(hunting): get_auth_request con 0x7f4ba40a8e50 auth_method 0 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4ba9a80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4ba9a80640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4ba9a80640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baa281640 10 monclient(hunting): get_auth_request con 0x7f4ba40a8a70 auth_method 0 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baa281640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baa281640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baa281640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baaa82640 10 monclient(hunting): get_auth_request con 0x7f4ba413f6d0 auth_method 0 2024-01-21T05:05:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baaa82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baaa82640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baaa82640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4ba9a80640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4ba9a80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4ba9a80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baa281640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baa281640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baa281640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baaa82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baaa82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.480+0000 7f4baaa82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.483+0000 7f4ba9a80640 10 monclient(hunting): handle_auth_done global_id 17433 payload 824 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.483+0000 7f4ba9a80640 1 -- 172.21.15.69:0/2596904798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ba40a8a70 msgr2=0x7f4ba40aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.483+0000 7f4ba9a80640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ba40a8a70 0x7f4ba40aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.483+0000 7f4ba9a80640 1 -- 172.21.15.69:0/2596904798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ba413f6d0 msgr2=0x7f4ba41d7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:46.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.483+0000 7f4ba9a80640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ba413f6d0 0x7f4ba41d7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.483+0000 7f4ba9a80640 10 monclient: _finish_hunting 0 2024-01-21T05:05:46.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.483+0000 7f4ba9a80640 1 monclient: found mon.noname-a 2024-01-21T05:05:46.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.483+0000 7f4ba9a80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:46.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.483+0000 7f4ba9a80640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:46.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.483+0000 7f4ba9a80640 1 -- 172.21.15.69:0/2596904798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba0009400 con 0x7f4ba40a8e50 2024-01-21T05:05:46.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.485+0000 7f4ba9a80640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ba40a8e50 0x7f4ba40ac440 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400c430 tx=0x7f4b9400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:46.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.485+0000 7f4b9affd640 10 client.?.objecter ms_handle_connect 0x7f4ba40a8e50 2024-01-21T05:05:46.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.485+0000 7f4b9affd640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 1 -- 172.21.15.69:0/2596904798 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94011790 con 0x7f4ba40a8e50 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 10 monclient: dump: 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 10 monclient: _finish_auth 0 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 10 monclient: _check_auth_tickets 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 1 -- 172.21.15.69:0/2596904798 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b940118f0 con 0x7f4ba40a8e50 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 1 -- 172.21.15.69:0/2596904798 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b94011ca0 con 0x7f4ba40a8e50 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4bac4e5640 5 monclient: authenticate success, global_id 17433 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4bac4e5640 10 monclient: _renew_subs 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4bac4e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4bac4e5640 1 -- 172.21.15.69:0/2596904798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba40ac980 con 0x7f4ba40a8e50 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4ba8a7e640 4 set_mon_vals no callback set 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4b9affd640 10 monclient: dump: 2024-01-21T05:05:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4bac4e5640 10 client.17433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4bac4e5640 10 monclient: _renew_subs 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4bac4e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4bac4e5640 1 -- 172.21.15.69:0/2596904798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba41d79a0 con 0x7f4ba40a8e50 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.488+0000 7f4bac4e5640 1 librados: init done 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.489+0000 7f4bac4e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.489+0000 7f4bac4e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:46.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.489+0000 7f4bac4e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:46.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.489+0000 7f4bac4e5640 1 -- 172.21.15.69:0/2596904798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba404e970 con 0x7f4ba40a8e50 2024-01-21T05:05:46.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.508+0000 7f4b9affd640 1 -- 172.21.15.69:0/2596904798 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4b9401c460 con 0x7f4ba40a8e50 2024-01-21T05:05:46.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.509+0000 7f4b9affd640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4b8806c860 0x7f4b8806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:46.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.509+0000 7f4baa281640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4b8806c860 0x7f4b8806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:46.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.509+0000 7f4baa281640 10 monclient: get_auth_request con 0x7f4b8806c860 auth_method 0 2024-01-21T05:05:46.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.510+0000 7f4b9affd640 10 client.17433.objecter ms_handle_connect 0x7f4b8806c860 2024-01-21T05:05:46.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.510+0000 7f4baa281640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4b8806c860 0x7f4b8806ed20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c005c00 tx=0x7f4b9c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:46.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.511+0000 7f4b9affd640 1 -- 172.21.15.69:0/2596904798 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1109..1109 src has 1032..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f4b94058b90 con 0x7f4ba40a8e50 2024-01-21T05:05:46.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.511+0000 7f4b9affd640 10 client.17433.objecter ms_dispatch 0x7f4ba4000b90 osd_map(1109..1109 src has 1032..1109) v4 2024-01-21T05:05:46.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.511+0000 7f4b9affd640 3 client.17433.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:46.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.511+0000 7f4b9affd640 3 client.17433.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:46.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.511+0000 7f4b9affd640 20 client.17433.objecter dump_active .. 0 homeless 2024-01-21T05:05:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.742+0000 7f4b9affd640 1 -- 172.21.15.69:0/2596904798 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4b94054ed0 con 0x7f4ba40a8e50 2024-01-21T05:05:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.742+0000 7f4b9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.742+0000 7f4b9affd640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:46.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.940+0000 7f4bac4e5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-01-21T05:05:46.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.940+0000 7f4bac4e5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T05:05:46.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.940+0000 7f4bac4e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:46.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.940+0000 7f4bac4e5640 1 -- 172.21.15.69:0/2596904798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ba41d6fa0 con 0x7f4ba40a8e50 2024-01-21T05:05:46.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.980+0000 7f4b9affd640 1 -- 172.21.15.69:0/2596904798 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1109) v1 ==== 74+0+17596 (secure 0 0 0) 0x7f4b940584f0 con 0x7f4ba40a8e50 2024-01-21T05:05:46.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.980+0000 7f4b9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-01-21T05:05:46.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.980+0000 7f4b9affd640 10 monclient: _finish_command 2 = system:0 2024-01-21T05:05:46.981 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T05:05:46.981 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1109,"fsid":"396a1cf2-e196-4a31-b42f-998c792d93aa","created":"2024-01-21T04:32:43.690443+0000","modified":"2024-01-21T05:05:44.268816+0000","last_up_change":"2024-01-21T04:33:38.448164+0000","last_in_change":"2024-01-21T04:32:48.763226+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":120,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-21T04:33:29.939892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-21T04:33:45.459402+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-01-21T04:34:44.677791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"598f21db-ec64-41f8-b46f-6808b345bf82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6825","nonce":335500148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6827","nonce":335500148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6831","nonce":335500148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":335500148},{"type":"v1","addr":"172.21.15.69:6829","nonce":335500148}]},"public_addr":"172.21.15.69:6825/335500148","cluster_addr":"172.21.15.69:6827/335500148","heartbeat_back_addr":"172.21.15.69:6831/335500148","heartbeat_front_addr":"172.21.15.69:6829/335500148","state":["exists","up"]},{"osd":1,"uuid":"f5228ed5-de32-48ef-9b87-809eab5bd25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6809","nonce":3868480563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6811","nonce":3868480563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6815","nonce":3868480563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3868480563},{"type":"v1","addr":"172.21.15.69:6813","nonce":3868480563}]},"public_addr":"172.21.15.69:6809/3868480563","cluster_addr":"172.21.15.69:6811/3868480563","heartbeat_back_addr":"172.21.15.69:6815/3868480563","heartbeat_front_addr":"172.21.15.69:6813/3868480563","state":["exists","up"]},{"osd":2,"uuid":"b7482f36-5fda-454d-9088-40ab951e3120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6817","nonce":3954012353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6819","nonce":3954012353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6823","nonce":3954012353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3954012353},{"type":"v1","addr":"172.21.15.69:6821","nonce":3954012353}]},"public_addr":"172.21.15.69:6817/3954012353","cluster_addr":"172.21.15.69:6819/3954012353","heartbeat_back_addr":"172.21.15.69:6823/3954012353","heartbeat_front_addr":"172.21.15.69:6821/3954012353","state":["exists","up"]},{"osd":3,"uuid":"429b547e-67bc-4d00-8f97-84d9cc4ad69a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":1105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6801","nonce":1086763384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6803","nonce":1086763384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6807","nonce":1086763384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1086763384},{"type":"v1","addr":"172.21.15.69:6805","nonce":1086763384}]},"public_addr":"172.21.15.69:6801/1086763384","cluster_addr":"172.21.15.69:6803/1086763384","heartbeat_back_addr":"172.21.15.69:6807/1086763384","heartbeat_front_addr":"172.21.15.69:6805/1086763384","state":["exists","up"]},{"osd":4,"uuid":"ca5cd910-9006-47e5-a83d-1d2533c24871","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6827","nonce":3897583001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6829","nonce":3897583001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6833","nonce":3897583001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":3897583001},{"type":"v1","addr":"172.21.15.144:6831","nonce":3897583001}]},"public_addr":"172.21.15.144:6827/3897583001","cluster_addr":"172.21.15.144:6829/3897583001","heartbeat_back_addr":"172.21.15.144:6833/3897583001","heartbeat_front_addr":"172.21.15.144:6831/3897583001","state":["exists","up"]},{"osd":5,"uuid":"e10f1b39-2112-44cf-8045-8b3bc316d07b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6803","nonce":1782989653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6805","nonce":1782989653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6809","nonce":1782989653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1782989653},{"type":"v1","addr":"172.21.15.144:6807","nonce":1782989653}]},"public_addr":"172.21.15.144:6803/1782989653","cluster_addr":"172.21.15.144:6805/1782989653","heartbeat_back_addr":"172.21.15.144:6809/1782989653","heartbeat_front_addr":"172.21.15.144:6807/1782989653","state":["exists","up"]},{"osd":6,"uuid":"a9b2ce7d-0ae7-4090-b33b-11403d70df1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6813","nonce":479183693}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6818","nonce":479183693}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6825","nonce":479183693}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":479183693},{"type":"v1","addr":"172.21.15.144:6822","nonce":479183693}]},"public_addr":"172.21.15.144:6813/479183693","cluster_addr":"172.21.15.144:6818/479183693","heartbeat_back_addr":"172.21.15.144:6825/479183693","heartbeat_front_addr":"172.21.15.144:6822/479183693","state":["exists","up"]},{"osd":7,"uuid":"ea6bd6a2-8108-43f6-a833-093408b6fc80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6811","nonce":14156089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6815","nonce":14156089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6821","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6823","nonce":14156089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6817","nonce":14156089},{"type":"v1","addr":"172.21.15.144:6819","nonce":14156089}]},"public_addr":"172.21.15.144:6811/14156089","cluster_addr":"172.21.15.144:6815/14156089","heartbeat_back_addr":"172.21.15.144:6823/14156089","heartbeat_front_addr":"172.21.15.144:6819/14156089","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:27.418834+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.672251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.150659+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.544731+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:26.167108+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:24.415555+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.272052+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-01-21T04:33:25.940928+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":5,"length":2}]},{"pool":22,"snaps":[{"begin":1,"length":4}]},{"pool":25,"snaps":[{"begin":1,"length":1}]},{"pool":75,"snaps":[{"begin":7,"length":2}]},{"pool":77,"snaps":[{"begin":1,"length":1}]},{"pool":86,"snaps":[{"begin":1,"length":1}]},{"pool":89,"snaps":[{"begin":1,"length":2}]},{"pool":96,"snaps":[{"begin":3,"length":1}]},{"pool":99,"snaps":[{"begin":2,"length":2}]},{"pool":110,"snaps":[{"begin":2,"length":3}]},{"pool":125,"snaps":[{"begin":3,"length":2}]},{"pool":127,"snaps":[{"begin":2,"length":2}]},{"pool":138,"snaps":[{"begin":4,"length":2}]},{"pool":141,"snaps":[{"begin":2,"length":1}]},{"pool":142,"snaps":[{"begin":3,"length":1}]},{"pool":145,"snaps":[{"begin":2,"length":3}]},{"pool":158,"snaps":[{"begin":4,"length":4}]},{"pool":229,"snaps":[{"begin":7,"length":1}]},{"pool":241,"snaps":[{"begin":8,"length":1}]},{"pool":247,"snaps":[{"begin":10,"length":1}]},{"pool":249,"snaps":[{"begin":11,"length":2}]},{"pool":262,"snaps":[{"begin":14,"length":1}]},{"pool":306,"snaps":[{"begin":2,"length":1}]},{"pool":307,"snaps":[{"begin":3,"length":1}]},{"pool":310,"snaps":[{"begin":4,"length":1}]},{"pool":314,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-21T05:05:46.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 10 librados: watch_flush enter 2024-01-21T05:05:46.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 10 librados: watch_flush exit 2024-01-21T05:05:46.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 20 client.17433.objecter shutdown clearing up homeless session... 2024-01-21T05:05:46.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 10 client.17433.objecter successfully canceled tick 2024-01-21T05:05:46.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 -- 172.21.15.69:0/2596904798 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4b8806c860 msgr2=0x7f4b8806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:46.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4b8806c860 0x7f4b8806ed20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c005c00 tx=0x7f4b9c009040 comp rx=0 tx=0).stop 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 10 monclient: shutdown 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 -- 172.21.15.69:0/2596904798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ba40a8e50 msgr2=0x7f4ba40ac440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ba40a8e50 0x7f4ba40ac440 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400c430 tx=0x7f4b9400c8f0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 -- 172.21.15.69:0/2596904798 shutdown_connections 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4b8806c860 0x7f4b8806ed20 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ba413f6d0 0x7f4ba41d7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ba40a8e50 0x7f4ba40ac440 unknown :-1 s=CLOSED pgs=3489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 --2- 172.21.15.69:0/2596904798 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4ba40a8a70 0x7f4ba40aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 -- 172.21.15.69:0/2596904798 >> 172.21.15.69:0/2596904798 conn(0x7f4ba4066f00 msgr2=0x7f4ba4139940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 -- 172.21.15.69:0/2596904798 shutdown_connections 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 -- 172.21.15.69:0/2596904798 wait complete. 2024-01-21T05:05:46.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:46.982+0000 7f4bac4e5640 1 librados: shutdown 2024-01-21T05:05:47.994 INFO:tasks.ceph:Scrubbing osd.0 2024-01-21T05:05:47.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-21T05:05:48.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.108+0000 7f5341f97640 10 monclient: get_monmap_and_config 2024-01-21T05:05:48.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.108+0000 7f5341f97640 10 monclient: build_initial_monmap 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.109+0000 7f5341f97640 10 monclient: monmap: 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:48.109994+0000 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:48.109994+0000 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:48.110 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.109+0000 7f5341f97640 1 Processor -- start 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.109+0000 7f5341f97640 1 -- start start 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.109+0000 7f5341f97640 10 monclient: init 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f533c0a8ab0 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f533c0a8e90 0x7f533c13f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 10 monclient(hunting): picked mon.noname-a con 0x7f533c0a8e90 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f533c0b2310 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c142dc0 con 0x7f533c0a8ab0 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c0aaa20 con 0x7f533c0b2310 2024-01-21T05:05:48.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f533c0aa820 con 0x7f533c0a8e90 2024-01-21T05:05:48.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:48.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f5341f97640 10 monclient(hunting): authenticate will time out at 3306.451580s 2024-01-21T05:05:48.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.110+0000 7f533affd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f533c0a8e90 0x7f533c13f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.111+0000 7f533bfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f533c0b2310 0x7f533c14c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:39386/0 (socket says 172.21.15.69:39386) 2024-01-21T05:05:48.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.111+0000 7f533bfff640 1 -- 172.21.15.69:0/3421983484 learned_addr learned my addr 172.21.15.69:0/3421983484 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T05:05:48.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.111+0000 7f533affd640 10 monclient(hunting): get_auth_request con 0x7f533c0a8e90 auth_method 0 2024-01-21T05:05:48.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.111+0000 7f533affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.111+0000 7f533affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.111+0000 7f533affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.111+0000 7f533affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.111+0000 7f533affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.111+0000 7f533affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533a7fc640 1 -- 172.21.15.69:0/3421983484 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683638935 0 0) 0x7f533c0aaa20 con 0x7f533c0b2310 2024-01-21T05:05:48.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533a7fc640 10 monclient(hunting): _negotiate 2024-01-21T05:05:48.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533a7fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533a7fc640 10 monclient(hunting): my global_id is 21547 2024-01-21T05:05:48.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533a7fc640 1 -- 172.21.15.69:0/3421983484 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5324003520 con 0x7f533c0b2310 2024-01-21T05:05:48.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533affd640 10 monclient(hunting): handle_auth_done global_id 17436 payload 274 2024-01-21T05:05:48.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533affd640 1 -- 172.21.15.69:0/3421983484 >> v1:172.21.15.144:6789/0 conn(0x7f533c0b2310 legacy=0x7f533c14c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533affd640 1 -- 172.21.15.69:0/3421983484 >> v1:172.21.15.69:6789/0 conn(0x7f533c0a8ab0 legacy=0x7f533c1426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:48.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533affd640 10 monclient: _finish_hunting 0 2024-01-21T05:05:48.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533affd640 1 monclient: found mon.noname-a 2024-01-21T05:05:48.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:48.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:48.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.113+0000 7f533affd640 1 -- 172.21.15.69:0/3421983484 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c0aad00 con 0x7f533c0a8e90 2024-01-21T05:05:48.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.115+0000 7f533affd640 1 --2- 172.21.15.69:0/3421983484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f533c0a8e90 0x7f533c13f240 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7f533c0a8710 tx=0x7f533002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73af8fbc478080c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:48.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 1 -- 172.21.15.69:0/3421983484 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533002e070 con 0x7f533c0a8e90 2024-01-21T05:05:48.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:48.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 10 monclient: dump: 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 10 monclient: _finish_auth 0 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 10 monclient: _check_auth_tickets 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:48.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 1 -- 172.21.15.69:0/3421983484 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53300315a0 con 0x7f533c0a8e90 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f5341f97640 5 monclient: authenticate success, global_id 17436 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f5341f97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f5341f97640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 1 -- 172.21.15.69:0/3421983484 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330031930 con 0x7f533c0a8e90 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f5341f97640 4 set_mon_vals no callback set 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.119+0000 7f533a7fc640 10 monclient: dump: 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.122 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.123 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.123 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.123 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.120+0000 7f5341f97640 10 monclient: shutdown 2024-01-21T05:05:48.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.120+0000 7f5341f97640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:48.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.120+0000 7f5341f97640 1 -- 172.21.15.69:0/3421983484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f533c0a8e90 msgr2=0x7f533c13f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.120+0000 7f5341f97640 1 --2- 172.21.15.69:0/3421983484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f533c0a8e90 0x7f533c13f240 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7f533c0a8710 tx=0x7f533002b840 comp rx=0 tx=0).stop 2024-01-21T05:05:48.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.120+0000 7f5341f97640 1 -- 172.21.15.69:0/3421983484 shutdown_connections 2024-01-21T05:05:48.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.120+0000 7f5341f97640 1 --2- 172.21.15.69:0/3421983484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f533c0a8e90 0x7f533c13f240 unknown :-1 s=CLOSED pgs=3490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.120+0000 7f5341f97640 1 -- 172.21.15.69:0/3421983484 >> 172.21.15.69:0/3421983484 conn(0x7f533c066f00 msgr2=0x7f533c0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:48.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.120+0000 7f5341f97640 1 -- 172.21.15.69:0/3421983484 shutdown_connections 2024-01-21T05:05:48.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.120+0000 7f5341f97640 1 -- 172.21.15.69:0/3421983484 wait complete. 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.121+0000 7f5341f97640 10 monclient: build_initial_monmap 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.121+0000 7f5341f97640 10 monclient: monmap: 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.121+0000 7f5341f97640 1 librados: starting msgr at 2024-01-21T05:05:48.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.121+0000 7f5341f97640 1 librados: starting objecter 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.121+0000 7f5341f97640 1 Processor -- start 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.121+0000 7f5341f97640 1 -- start start 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.121+0000 7f5341f97640 1 librados: setting wanted keys 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.121+0000 7f5341f97640 1 librados: calling monclient init 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.121+0000 7f5341f97640 10 monclient: init 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f533c0a8ab0 0x7f533c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 10 monclient(hunting): picked mon.noname-a con 0x7f533c0a8ab0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f533c0a8e90 0x7f533c0adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 10 monclient(hunting): picked mon.noname-c con 0x7f533c0a8e90 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f533c0b2310 0x7f533c0b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 10 monclient(hunting): picked mon.noname-b con 0x7f533c0b2310 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:48.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c0b00b0 con 0x7f533c0a8ab0 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c0b16d0 con 0x7f533c0b2310 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533c0ae3e0 con 0x7f533c0a8e90 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.122+0000 7f5341f97640 10 monclient(hunting): authenticate will time out at 3306.463736s 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533affd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f533c0a8e90 0x7f533c0adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f533c0a8ab0 0x7f533c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533bfff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f533c0b2310 0x7f533c0b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f533c0a8ab0 0x7f533c0aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41586/0 (socket says 172.21.15.69:41586) 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533b7fe640 1 -- 172.21.15.69:0/2043289979 learned_addr learned my addr 172.21.15.69:0/2043289979 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:48.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533b7fe640 10 monclient(hunting): get_auth_request con 0x7f533c0a8ab0 auth_method 0 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533affd640 10 monclient(hunting): get_auth_request con 0x7f533c0a8e90 auth_method 0 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533bfff640 10 monclient(hunting): get_auth_request con 0x7f533c0b2310 auth_method 0 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.123+0000 7f533affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.124+0000 7f533affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.124+0000 7f533bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.124+0000 7f533bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.124+0000 7f533bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.126+0000 7f533b7fe640 10 monclient(hunting): handle_auth_done global_id 17439 payload 824 2024-01-21T05:05:48.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.126+0000 7f533b7fe640 1 -- 172.21.15.69:0/2043289979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f533c0a8e90 msgr2=0x7f533c0adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.126+0000 7f533b7fe640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f533c0a8e90 0x7f533c0adea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.126+0000 7f533b7fe640 1 -- 172.21.15.69:0/2043289979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f533c0b2310 msgr2=0x7f533c0b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.126+0000 7f533b7fe640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f533c0b2310 0x7f533c0b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.126+0000 7f533b7fe640 10 monclient: _finish_hunting 0 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.126+0000 7f533b7fe640 1 monclient: found mon.noname-a 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.126+0000 7f533b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.126+0000 7f533b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.126+0000 7f533b7fe640 1 -- 172.21.15.69:0/2043289979 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5330009400 con 0x7f533c0a8ab0 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.129+0000 7f533b7fe640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f533c0a8ab0 0x7f533c0aeba0 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7f532c00c410 tx=0x7f532c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.129+0000 7f5323fff640 10 client.?.objecter ms_handle_connect 0x7f533c0a8ab0 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.129+0000 7f5323fff640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5323fff640 1 -- 172.21.15.69:0/2043289979 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c0035f0 con 0x7f533c0a8ab0 2024-01-21T05:05:48.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5323fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5323fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5323fff640 10 monclient: dump: 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5323fff640 10 monclient: _finish_auth 0 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5323fff640 10 monclient: _check_auth_tickets 2024-01-21T05:05:48.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5323fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5341f97640 5 monclient: authenticate success, global_id 17439 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5341f97640 10 monclient: _renew_subs 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5341f97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5341f97640 1 -- 172.21.15.69:0/2043289979 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533c0b0720 con 0x7f533c0a8ab0 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5341f97640 10 client.17439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5341f97640 10 monclient: _renew_subs 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5341f97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5341f97640 1 -- 172.21.15.69:0/2043289979 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533c0b13d0 con 0x7f533c0a8ab0 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.131+0000 7f5341f97640 1 librados: init done 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.132+0000 7f5341f97640 10 client.17439.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.132+0000 7f5341f97640 15 client.17439.objecter _session_command_op_assign -1 1 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.132+0000 7f5341f97640 20 client.17439.objecter _get_session osd=-1 returning homeless 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.132+0000 7f5341f97640 10 client.17439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.132+0000 7f5341f97640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:48.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.132+0000 7f5341f97640 1 -- 172.21.15.69:0/2043289979 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f533c142190 con 0x7f533c0a8ab0 2024-01-21T05:05:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.152+0000 7f5323fff640 1 -- 172.21.15.69:0/2043289979 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f532c001120 con 0x7f533c0a8ab0 2024-01-21T05:05:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.152+0000 7f5323fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.152+0000 7f5323fff640 1 -- 172.21.15.69:0/2043289979 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f532c007220 con 0x7f533c0a8ab0 2024-01-21T05:05:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.152+0000 7f5323fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.152+0000 7f5323fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.152+0000 7f5323fff640 10 monclient: dump: 2024-01-21T05:05:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.154 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.152+0000 7f5339ffb640 4 set_mon_vals no callback set 2024-01-21T05:05:48.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.153+0000 7f5323fff640 1 -- 172.21.15.69:0/2043289979 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f532c007440 con 0x7f533c0a8ab0 2024-01-21T05:05:48.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.153+0000 7f5323fff640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f532806c860 0x7f532806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.153+0000 7f533affd640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f532806c860 0x7f532806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.154+0000 7f533affd640 10 monclient: get_auth_request con 0x7f532806c860 auth_method 0 2024-01-21T05:05:48.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.154+0000 7f5323fff640 10 client.17439.objecter ms_handle_connect 0x7f532806c860 2024-01-21T05:05:48.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.154+0000 7f533affd640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f532806c860 0x7f532806ed20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5330036000 tx=0x7f533003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 1 -- 172.21.15.69:0/2043289979 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1109..1109 src has 1032..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f532c058030 con 0x7f533c0a8ab0 2024-01-21T05:05:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 10 client.17439.objecter ms_dispatch 0x7f533c000b90 osd_map(1109..1109 src has 1032..1109) v4 2024-01-21T05:05:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 3 client.17439.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 3 client.17439.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 10 client.17439.objecter checking command 1 2024-01-21T05:05:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] conn(0x7f5328074550 0x7f53280769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 20 client.17439.objecter _get_session s=0x7f5328073e80 osd=0 3 2024-01-21T05:05:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 20 client.17439.objecter put_session s=0x7f5328073e80 osd=0 3 2024-01-21T05:05:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 15 client.17439.objecter _session_command_op_remove -1 1 2024-01-21T05:05:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 20 client.17439.objecter _get_session s=0x7f5328073e80 osd=0 3 2024-01-21T05:05:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 20 client.17439.objecter get_session s=0x7f5328073e80 osd=0 3 2024-01-21T05:05:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 15 client.17439.objecter _session_command_op_assign 0 1 2024-01-21T05:05:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 20 client.17439.objecter put_session s=0x7f5328073e80 osd=0 4 2024-01-21T05:05:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 10 client.17439.objecter _send_command 1 2024-01-21T05:05:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 1 -- 172.21.15.69:0/2043289979 --> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53280770c0 con 0x7f5328074550 2024-01-21T05:05:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 20 client.17439.objecter dump_active .. 0 homeless 2024-01-21T05:05:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 1 -- 172.21.15.69:0/2043289979 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=1109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f532c054370 con 0x7f533c0a8ab0 2024-01-21T05:05:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.156+0000 7f5323fff640 10 monclient: handle_get_version_reply finishing 1 version 1109 2024-01-21T05:05:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.157+0000 7f533bfff640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] conn(0x7f5328074550 0x7f53280769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.157+0000 7f533bfff640 10 monclient: get_auth_request con 0x7f5328074550 auth_method 0 2024-01-21T05:05:48.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.159+0000 7f5323fff640 10 client.17439.objecter ms_handle_connect 0x7f5328074550 2024-01-21T05:05:48.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.159+0000 7f533bfff640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] conn(0x7f5328074550 0x7f53280769f0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:48.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.185+0000 7f5323fff640 1 -- 172.21.15.69:0/2043289979 <== osd.0 v2:172.21.15.69:6824/335500148 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f53280770c0 con 0x7f5328074550 2024-01-21T05:05:48.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.185+0000 7f5323fff640 10 client.17439.objecter ms_dispatch 0x7f533c000b90 command_reply(tid 1: 0 ) v1 2024-01-21T05:05:48.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.185+0000 7f5323fff640 10 client.17439.objecter _finish_command 1 = system:0 2024-01-21T05:05:48.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.185+0000 7f5323fff640 20 client.17439.objecter put_session s=0x7f5328073e80 osd=0 4 2024-01-21T05:05:48.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.185+0000 7f5323fff640 15 client.17439.objecter _session_command_op_remove 0 1 2024-01-21T05:05:48.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.198+0000 7f5341f97640 10 client.17439.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T05:05:48.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.198+0000 7f5341f97640 15 client.17439.objecter _session_command_op_assign -1 2 2024-01-21T05:05:48.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.198+0000 7f5341f97640 20 client.17439.objecter _get_session s=0x7f5328073e80 osd=0 3 2024-01-21T05:05:48.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.198+0000 7f5341f97640 20 client.17439.objecter put_session s=0x7f5328073e80 osd=0 3 2024-01-21T05:05:48.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.198+0000 7f5341f97640 20 client.17439.objecter _get_session s=0x7f5328073e80 osd=0 3 2024-01-21T05:05:48.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.198+0000 7f5341f97640 15 client.17439.objecter _session_command_op_remove -1 2 2024-01-21T05:05:48.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.198+0000 7f5341f97640 20 client.17439.objecter get_session s=0x7f5328073e80 osd=0 3 2024-01-21T05:05:48.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.198+0000 7f5341f97640 15 client.17439.objecter _session_command_op_assign 0 2 2024-01-21T05:05:48.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.198+0000 7f5341f97640 20 client.17439.objecter put_session s=0x7f5328073e80 osd=0 4 2024-01-21T05:05:48.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.198+0000 7f5341f97640 10 client.17439.objecter _send_command 2 2024-01-21T05:05:48.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.198+0000 7f5341f97640 1 -- 172.21.15.69:0/2043289979 --> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f533c04e970 con 0x7f5328074550 2024-01-21T05:05:48.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5323fff640 1 -- 172.21.15.69:0/2043289979 <== osd.0 v2:172.21.15.69:6824/335500148 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f533c04e970 con 0x7f5328074550 2024-01-21T05:05:48.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5323fff640 10 client.17439.objecter ms_dispatch 0x7f533c000b90 command_reply(tid 2: 0 ) v1 2024-01-21T05:05:48.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5323fff640 10 client.17439.objecter _finish_command 2 = system:0 2024-01-21T05:05:48.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5323fff640 20 client.17439.objecter put_session s=0x7f5328073e80 osd=0 4 2024-01-21T05:05:48.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5323fff640 15 client.17439.objecter _session_command_op_remove 0 2 2024-01-21T05:05:48.212 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-21T05:05:48.213 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T05:05:48.213 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-21T05:05:48.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5341f97640 10 librados: watch_flush enter 2024-01-21T05:05:48.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5341f97640 10 librados: watch_flush exit 2024-01-21T05:05:48.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5341f97640 10 client.17439.objecter close_session for osd.0 2024-01-21T05:05:48.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5341f97640 1 -- 172.21.15.69:0/2043289979 >> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] conn(0x7f5328074550 msgr2=0x7f53280769f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5341f97640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.69:6824/335500148,v1:172.21.15.69:6825/335500148] conn(0x7f5328074550 0x7f53280769f0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5341f97640 20 client.17439.objecter put_session s=0x7f5328073e80 osd=0 1 2024-01-21T05:05:48.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5341f97640 20 client.17439.objecter shutdown clearing up homeless session... 2024-01-21T05:05:48.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5341f97640 10 client.17439.objecter successfully canceled tick 2024-01-21T05:05:48.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5341f97640 1 -- 172.21.15.69:0/2043289979 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f532806c860 msgr2=0x7f532806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.211+0000 7f5341f97640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f532806c860 0x7f532806ed20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5330036000 tx=0x7f533003d040 comp rx=0 tx=0).stop 2024-01-21T05:05:48.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.212+0000 7f5341f97640 10 monclient: shutdown 2024-01-21T05:05:48.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.212+0000 7f5341f97640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:48.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.212+0000 7f5341f97640 1 -- 172.21.15.69:0/2043289979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f533c0a8ab0 msgr2=0x7f533c0aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.212+0000 7f5341f97640 1 --2- 172.21.15.69:0/2043289979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f533c0a8ab0 0x7f533c0aeba0 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7f532c00c410 tx=0x7f532c00c8d0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.212+0000 7f533bfff640 1 -- 172.21.15.69:0/2043289979 reap_dead start 2024-01-21T05:05:48.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.212+0000 7f5341f97640 1 -- 172.21.15.69:0/2043289979 shutdown_connections 2024-01-21T05:05:48.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.212+0000 7f5341f97640 1 -- 172.21.15.69:0/2043289979 >> 172.21.15.69:0/2043289979 conn(0x7f533c066f00 msgr2=0x7f533c13fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:48.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.212+0000 7f5341f97640 1 -- 172.21.15.69:0/2043289979 shutdown_connections 2024-01-21T05:05:48.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.212+0000 7f5341f97640 1 -- 172.21.15.69:0/2043289979 wait complete. 2024-01-21T05:05:48.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.212+0000 7f5341f97640 1 librados: shutdown 2024-01-21T05:05:48.223 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-21T05:05:48.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.326+0000 7f6f45ebb640 10 monclient: get_monmap_and_config 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.326+0000 7f6f45ebb640 10 monclient: build_initial_monmap 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.326+0000 7f6f45ebb640 10 monclient: monmap: 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:48.327209+0000 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:48.327209+0000 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:48.327 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.326+0000 7f6f45ebb640 1 Processor -- start 2024-01-21T05:05:48.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 1 -- start start 2024-01-21T05:05:48.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 10 monclient: init 2024-01-21T05:05:48.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:48.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6f400a8b20 0x7f6f40145b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 10 monclient(hunting): picked mon.noname-c con 0x7f6f400a8b20 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f400a9d10 0x7f6f4013f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 10 monclient(hunting): picked mon.noname-a con 0x7f6f400a9d10 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f400b2310 0x7f6f4013f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 10 monclient(hunting): picked mon.noname-b con 0x7f6f400b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f401456f0 con 0x7f6f400a9d10 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f400a98c0 con 0x7f6f400a8b20 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4013fb00 con 0x7f6f400b2310 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:48.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.327+0000 7f6f45ebb640 10 monclient(hunting): authenticate will time out at 3306.668900s 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f400b2310 0x7f6f4013f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3effd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f400a9d10 0x7f6f4013f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3f7fe640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f6f400a8b20 0x7f6f40145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3ffff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f400b2310 0x7f6f4013f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:48538/0 (socket says 172.21.15.69:48538) 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3ffff640 1 -- 172.21.15.69:0/1591480144 learned_addr learned my addr 172.21.15.69:0/1591480144 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3ffff640 10 monclient(hunting): get_auth_request con 0x7f6f400b2310 auth_method 0 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3effd640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f400a9d10 0x7f6f4013f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41594/0 (socket says 172.21.15.69:41594) 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3effd640 10 monclient(hunting): get_auth_request con 0x7f6f400a9d10 auth_method 0 2024-01-21T05:05:48.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3effd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3f7fe640 10 monclient(hunting): get_auth_request con 0x7f6f400a8b20 auth_method 0 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.328+0000 7f6f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.329+0000 7f6f3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.329+0000 7f6f3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.329+0000 7f6f3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.329+0000 7f6f3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.329+0000 7f6f3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.329+0000 7f6f3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.329+0000 7f6f3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.329+0000 7f6f3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.329+0000 7f6f3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.330+0000 7f6f3effd640 10 monclient(hunting): handle_auth_done global_id 17442 payload 274 2024-01-21T05:05:48.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.330+0000 7f6f3effd640 1 -- 172.21.15.69:0/1591480144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f400b2310 msgr2=0x7f6f4013f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:48.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.330+0000 7f6f3effd640 1 --2- 172.21.15.69:0/1591480144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f400b2310 0x7f6f4013f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.330+0000 7f6f3effd640 1 -- 172.21.15.69:0/1591480144 >> v2:172.21.15.144:3300/0 conn(0x7f6f400a8b20 msgr2=0x7f6f40145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:48.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.330+0000 7f6f3effd640 1 --2- 172.21.15.69:0/1591480144 >> v2:172.21.15.144:3300/0 conn(0x7f6f400a8b20 0x7f6f40145b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.330+0000 7f6f3effd640 10 monclient: _finish_hunting 0 2024-01-21T05:05:48.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.330+0000 7f6f3effd640 1 monclient: found mon.noname-a 2024-01-21T05:05:48.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.330+0000 7f6f3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:48.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.330+0000 7f6f3effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:48.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.330+0000 7f6f3effd640 1 -- 172.21.15.69:0/1591480144 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f401460c0 con 0x7f6f400a9d10 2024-01-21T05:05:48.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.332+0000 7f6f3effd640 1 --2- 172.21.15.69:0/1591480144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f400a9d10 0x7f6f4013f080 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7f6f400a8600 tx=0x7f6f3002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9ff6eb959b4702 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:48.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f3e7fc640 1 -- 172.21.15.69:0/1591480144 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3002e070 con 0x7f6f400a9d10 2024-01-21T05:05:48.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f3e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:48.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f3e7fc640 10 monclient: dump: 2024-01-21T05:05:48.338 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.338 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.338 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.338 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.338 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.338 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.338 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.338 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f3e7fc640 10 monclient: _finish_auth 0 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f3e7fc640 10 monclient: _check_auth_tickets 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f3e7fc640 1 -- 172.21.15.69:0/1591480144 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f300315a0 con 0x7f6f400a9d10 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f3e7fc640 1 -- 172.21.15.69:0/1591480144 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30031930 con 0x7f6f400a9d10 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f45ebb640 5 monclient: authenticate success, global_id 17442 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f45ebb640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f45ebb640 4 set_mon_vals no callback set 2024-01-21T05:05:48.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.335+0000 7f6f3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f3e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f3e7fc640 10 monclient: dump: 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.340 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f45ebb640 10 monclient: shutdown 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f45ebb640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f45ebb640 1 -- 172.21.15.69:0/1591480144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f400a9d10 msgr2=0x7f6f4013f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f45ebb640 1 --2- 172.21.15.69:0/1591480144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f400a9d10 0x7f6f4013f080 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7f6f400a8600 tx=0x7f6f3002b840 comp rx=0 tx=0).stop 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f45ebb640 1 -- 172.21.15.69:0/1591480144 shutdown_connections 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f45ebb640 1 --2- 172.21.15.69:0/1591480144 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f400b2310 0x7f6f4013f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f45ebb640 1 --2- 172.21.15.69:0/1591480144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f400a9d10 0x7f6f4013f080 unknown :-1 s=CLOSED pgs=3492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f45ebb640 1 --2- 172.21.15.69:0/1591480144 >> v2:172.21.15.144:3300/0 conn(0x7f6f400a8b20 0x7f6f40145b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f45ebb640 1 -- 172.21.15.69:0/1591480144 >> 172.21.15.69:0/1591480144 conn(0x7f6f40066fb0 msgr2=0x7f6f400b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f45ebb640 1 -- 172.21.15.69:0/1591480144 shutdown_connections 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.336+0000 7f6f45ebb640 1 -- 172.21.15.69:0/1591480144 wait complete. 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.337+0000 7f6f45ebb640 10 monclient: build_initial_monmap 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.337+0000 7f6f45ebb640 10 monclient: monmap: 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:48.341 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:48.342 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:48.342 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:48.342 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:48.342 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.342 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:48.342 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:48.342 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:48.342 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.337+0000 7f6f45ebb640 1 librados: starting msgr at 2024-01-21T05:05:48.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.337+0000 7f6f45ebb640 1 librados: starting objecter 2024-01-21T05:05:48.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.337+0000 7f6f45ebb640 1 Processor -- start 2024-01-21T05:05:48.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 1 -- start start 2024-01-21T05:05:48.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 1 librados: setting wanted keys 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 1 librados: calling monclient init 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 10 monclient: init 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f400a8b20 0x7f6f400b6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 10 monclient(hunting): picked mon.noname-b con 0x7f6f400a8b20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f400a9d10 0x7f6f401d79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 10 monclient(hunting): picked mon.noname-c con 0x7f6f400a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f400b2310 0x7f6f401d7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 10 monclient(hunting): picked mon.noname-a con 0x7f6f400b2310 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f40143880 con 0x7f6f400b2310 2024-01-21T05:05:48.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f401d74d0 con 0x7f6f400a8b20 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f400b7510 con 0x7f6f400a9d10 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.338+0000 7f6f45ebb640 10 monclient(hunting): authenticate will time out at 3306.679898s 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f400b2310 0x7f6f401d7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3f7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f400a8b20 0x7f6f400b6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3effd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f400a9d10 0x7f6f401d79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f400b2310 0x7f6f401d7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41602/0 (socket says 172.21.15.69:41602) 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3ffff640 1 -- 172.21.15.69:0/469252138 learned_addr learned my addr 172.21.15.69:0/469252138 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3ffff640 10 monclient(hunting): get_auth_request con 0x7f6f400b2310 auth_method 0 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3ffff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3effd640 10 monclient(hunting): get_auth_request con 0x7f6f400a9d10 auth_method 0 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3effd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3effd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3f7fe640 10 monclient(hunting): get_auth_request con 0x7f6f400a8b20 auth_method 0 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3f7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.339+0000 7f6f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.340+0000 7f6f3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.340+0000 7f6f3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.340+0000 7f6f3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.340+0000 7f6f3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.340+0000 7f6f3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.340+0000 7f6f3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.340+0000 7f6f3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.340+0000 7f6f3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.340+0000 7f6f3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.342+0000 7f6f3f7fe640 10 monclient(hunting): handle_auth_done global_id 21556 payload 824 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.342+0000 7f6f3f7fe640 1 -- 172.21.15.69:0/469252138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f400a9d10 msgr2=0x7f6f401d79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.342+0000 7f6f3f7fe640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f400a9d10 0x7f6f401d79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.342+0000 7f6f3f7fe640 1 -- 172.21.15.69:0/469252138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f400b2310 msgr2=0x7f6f401d7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.342+0000 7f6f3f7fe640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f400b2310 0x7f6f401d7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.342+0000 7f6f3f7fe640 10 monclient: _finish_hunting 0 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.342+0000 7f6f3f7fe640 1 monclient: found mon.noname-b 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.342+0000 7f6f3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:48.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.342+0000 7f6f3f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:05:48.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.342+0000 7f6f3f7fe640 1 -- 172.21.15.69:0/469252138 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f30009400 con 0x7f6f400a8b20 2024-01-21T05:05:48.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.344+0000 7f6f3f7fe640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f400a8b20 0x7f6f400b6fd0 secure :-1 s=READY pgs=5219 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400e850 tx=0x7f6f3400ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:48.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.344+0000 7f6f27fff640 10 client.?.objecter ms_handle_connect 0x7f6f400a8b20 2024-01-21T05:05:48.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.344+0000 7f6f27fff640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:48.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 1 -- 172.21.15.69:0/469252138 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f340095b0 con 0x7f6f400a8b20 2024-01-21T05:05:48.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:48.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 10 monclient: dump: 2024-01-21T05:05:48.351 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.351 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 10 monclient: _finish_auth 0 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 10 monclient: _check_auth_tickets 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 1 -- 172.21.15.69:0/469252138 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3400c940 con 0x7f6f400a8b20 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f45ebb640 5 monclient: authenticate success, global_id 21556 2024-01-21T05:05:48.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f45ebb640 10 monclient: _renew_subs 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f45ebb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f45ebb640 1 -- 172.21.15.69:0/469252138 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f400b6430 con 0x7f6f400a8b20 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 1 -- 172.21.15.69:0/469252138 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3400ccd0 con 0x7f6f400a8b20 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f27fff640 10 monclient: dump: 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.353 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f3dffb640 4 set_mon_vals no callback set 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f45ebb640 10 client.21556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f45ebb640 10 monclient: _renew_subs 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f45ebb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f45ebb640 1 -- 172.21.15.69:0/469252138 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f40142720 con 0x7f6f400a8b20 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.348+0000 7f6f45ebb640 1 librados: init done 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.349+0000 7f6f45ebb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.349+0000 7f6f45ebb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.349+0000 7f6f45ebb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:48.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.349+0000 7f6f45ebb640 1 -- 172.21.15.69:0/469252138 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4004e970 con 0x7f6f400a8b20 2024-01-21T05:05:48.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.368+0000 7f6f27fff640 1 -- 172.21.15.69:0/469252138 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6f34013470 con 0x7f6f400a8b20 2024-01-21T05:05:48.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.368+0000 7f6f27fff640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6f2c06c8b0 0x7f6f2c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.369+0000 7f6f3effd640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6f2c06c8b0 0x7f6f2c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.369+0000 7f6f3effd640 10 monclient: get_auth_request con 0x7f6f2c06c8b0 auth_method 0 2024-01-21T05:05:48.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.369+0000 7f6f3effd640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6f2c06c8b0 0x7f6f2c06ed70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6f30036000 tx=0x7f6f3003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:48.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.369+0000 7f6f27fff640 10 client.21556.objecter ms_handle_connect 0x7f6f2c06c8b0 2024-01-21T05:05:48.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.598+0000 7f6f27fff640 1 -- 172.21.15.69:0/469252138 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1032..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f6f3400ce30 con 0x7f6f400a8b20 2024-01-21T05:05:48.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.598+0000 7f6f27fff640 10 client.21556.objecter ms_dispatch 0x7f6f40000b90 osd_map(1109..1109 src has 1032..1109) v4 2024-01-21T05:05:48.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.598+0000 7f6f27fff640 3 client.21556.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:48.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.598+0000 7f6f27fff640 3 client.21556.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:48.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.598+0000 7f6f27fff640 20 client.21556.objecter dump_active .. 0 homeless 2024-01-21T05:05:48.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.608+0000 7f6f27fff640 1 -- 172.21.15.69:0/469252138 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6f34054e70 con 0x7f6f400a8b20 2024-01-21T05:05:48.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.608+0000 7f6f27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:48.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.608+0000 7f6f27fff640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:48.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.781+0000 7f6f45ebb640 1 -- 172.21.15.69:0/469252138 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f401439f0 con 0x7f6f2c06c8b0 2024-01-21T05:05:48.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.782+0000 7f6f27fff640 1 -- 172.21.15.69:0/469252138 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6f401439f0 con 0x7f6f2c06c8b0 2024-01-21T05:05:48.783 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 0 to deep-scrub 2024-01-21T05:05:48.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 10 librados: watch_flush enter 2024-01-21T05:05:48.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 10 librados: watch_flush exit 2024-01-21T05:05:48.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 20 client.21556.objecter shutdown clearing up homeless session... 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 10 client.21556.objecter successfully canceled tick 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 -- 172.21.15.69:0/469252138 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6f2c06c8b0 msgr2=0x7f6f2c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6f2c06c8b0 0x7f6f2c06ed70 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6f30036000 tx=0x7f6f3003d040 comp rx=0 tx=0).stop 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 10 monclient: shutdown 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 -- 172.21.15.69:0/469252138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f400a8b20 msgr2=0x7f6f400b6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f400a8b20 0x7f6f400b6fd0 secure :-1 s=READY pgs=5219 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400e850 tx=0x7f6f3400ed10 comp rx=0 tx=0).stop 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 -- 172.21.15.69:0/469252138 shutdown_connections 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f6f2c06c8b0 0x7f6f2c06ed70 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f400b2310 0x7f6f401d7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f6f400a9d10 0x7f6f401d79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 --2- 172.21.15.69:0/469252138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f400a8b20 0x7f6f400b6fd0 unknown :-1 s=CLOSED pgs=5219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 -- 172.21.15.69:0/469252138 >> 172.21.15.69:0/469252138 conn(0x7f6f40066fb0 msgr2=0x7f6f4013b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 -- 172.21.15.69:0/469252138 shutdown_connections 2024-01-21T05:05:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 -- 172.21.15.69:0/469252138 wait complete. 2024-01-21T05:05:48.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.784+0000 7f6f45ebb640 1 librados: shutdown 2024-01-21T05:05:48.796 INFO:tasks.ceph:Scrubbing osd.1 2024-01-21T05:05:48.796 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-21T05:05:48.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.898+0000 7fdd40ada640 10 monclient: get_monmap_and_config 2024-01-21T05:05:48.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.898+0000 7fdd40ada640 10 monclient: build_initial_monmap 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.899+0000 7fdd40ada640 10 monclient: monmap: 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:48.900003+0000 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:48.900003+0000 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:48.901 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.899+0000 7fdd40ada640 1 Processor -- start 2024-01-21T05:05:48.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.899+0000 7fdd40ada640 1 -- start start 2024-01-21T05:05:48.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.899+0000 7fdd40ada640 10 monclient: init 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd3c0ae300 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c070a60 0x7fdd3c13ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 10 monclient(hunting): picked mon.noname-b con 0x7fdd3c070a60 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd3c13f250 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3c071bc0 con 0x7fdd3c13f250 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3c06fb30 con 0x7fdd3c0ae300 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c0ae6e0 con 0x7fdd3c070a60 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd40ada640 10 monclient(hunting): authenticate will time out at 3307.241530s 2024-01-21T05:05:48.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd33fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c070a60 0x7fdd3c13ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd3ad9d640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdd3c13f250 0x7fdd3c149520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50212/0 (socket says 172.21.15.69:50212) 2024-01-21T05:05:48.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.900+0000 7fdd3ad9d640 1 -- 172.21.15.69:0/2475431383 learned_addr learned my addr 172.21.15.69:0/2475431383 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T05:05:48.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.901+0000 7fdd33fff640 10 monclient(hunting): get_auth_request con 0x7fdd3c070a60 auth_method 0 2024-01-21T05:05:48.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.901+0000 7fdd33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.901+0000 7fdd33fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.901+0000 7fdd33fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.901+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.901+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.901+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 1 -- 172.21.15.69:0/2475431383 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970810807 0 0) 0x7fdd3c06fb30 con 0x7fdd3c0ae300 2024-01-21T05:05:48.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 10 monclient(hunting): _negotiate 2024-01-21T05:05:48.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 10 monclient(hunting): my global_id is 21559 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 1 -- 172.21.15.69:0/2475431383 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd28003520 con 0x7fdd3c0ae300 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 1 -- 172.21.15.69:0/2475431383 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155655773 0 0) 0x7fdd3c071bc0 con 0x7fdd3c13f250 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 10 monclient(hunting): _negotiate 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 10 monclient(hunting): my global_id is 17448 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd39d9b640 1 -- 172.21.15.69:0/2475431383 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd3c06fb30 con 0x7fdd3c13f250 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd33fff640 10 monclient(hunting): handle_auth_done global_id 18827 payload 274 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd33fff640 1 -- 172.21.15.69:0/2475431383 >> v1:172.21.15.144:6789/0 conn(0x7fdd3c0ae300 legacy=0x7fdd3c13bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd33fff640 1 -- 172.21.15.69:0/2475431383 >> v1:172.21.15.69:6789/0 conn(0x7fdd3c13f250 legacy=0x7fdd3c149520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd33fff640 10 monclient: _finish_hunting 0 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd33fff640 1 monclient: found mon.noname-b 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd33fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T05:05:48.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.903+0000 7fdd33fff640 1 -- 172.21.15.69:0/2475431383 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c071e10 con 0x7fdd3c070a60 2024-01-21T05:05:48.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.907+0000 7fdd33fff640 1 --2- 172.21.15.69:0/2475431383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c070a60 0x7fdd3c13ed10 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7fdd240097d0 tx=0x7fdd2402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=235a0a15cb309246 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:48.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 1 -- 172.21.15.69:0/2475431383 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24031800 con 0x7fdd3c070a60 2024-01-21T05:05:48.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:48.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 10 monclient: dump: 2024-01-21T05:05:48.912 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 10 monclient: _finish_auth 0 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 10 monclient: _check_auth_tickets 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 1 -- 172.21.15.69:0/2475431383 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd24031960 con 0x7fdd3c070a60 2024-01-21T05:05:48.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 1 -- 172.21.15.69:0/2475431383 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24031cf0 con 0x7fdd3c070a60 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd40ada640 5 monclient: authenticate success, global_id 18827 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd40ada640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd40ada640 4 set_mon_vals no callback set 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.911+0000 7fdd39d9b640 10 monclient: dump: 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.914 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.915 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.915 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.915 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.915 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.912+0000 7fdd40ada640 10 monclient: shutdown 2024-01-21T05:05:48.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.912+0000 7fdd40ada640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:48.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.912+0000 7fdd40ada640 1 -- 172.21.15.69:0/2475431383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c070a60 msgr2=0x7fdd3c13ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.912+0000 7fdd40ada640 1 --2- 172.21.15.69:0/2475431383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c070a60 0x7fdd3c13ed10 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7fdd240097d0 tx=0x7fdd2402b8a0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.912+0000 7fdd40ada640 1 -- 172.21.15.69:0/2475431383 shutdown_connections 2024-01-21T05:05:48.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.912+0000 7fdd40ada640 1 --2- 172.21.15.69:0/2475431383 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c070a60 0x7fdd3c13ed10 unknown :-1 s=CLOSED pgs=3323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.912+0000 7fdd40ada640 1 -- 172.21.15.69:0/2475431383 >> 172.21.15.69:0/2475431383 conn(0x7fdd3c0b3690 msgr2=0x7fdd3c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:48.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.912+0000 7fdd40ada640 1 -- 172.21.15.69:0/2475431383 shutdown_connections 2024-01-21T05:05:48.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.912+0000 7fdd40ada640 1 -- 172.21.15.69:0/2475431383 wait complete. 2024-01-21T05:05:48.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.913+0000 7fdd40ada640 10 monclient: build_initial_monmap 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.913+0000 7fdd40ada640 10 monclient: monmap: 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.913+0000 7fdd40ada640 1 librados: starting msgr at 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.913+0000 7fdd40ada640 1 librados: starting objecter 2024-01-21T05:05:48.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.913+0000 7fdd40ada640 1 Processor -- start 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.913+0000 7fdd40ada640 1 -- start start 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.913+0000 7fdd40ada640 1 librados: setting wanted keys 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.913+0000 7fdd40ada640 1 librados: calling monclient init 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.913+0000 7fdd40ada640 10 monclient: init 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c070a60 0x7fdd3c142440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 10 monclient(hunting): picked mon.noname-c con 0x7fdd3c070a60 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd3c0ae300 0x7fdd3c13dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 10 monclient(hunting): picked mon.noname-b con 0x7fdd3c0ae300 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c13f250 0x7fdd3c1d8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 10 monclient(hunting): picked mon.noname-a con 0x7fdd3c13f250 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:48.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c13eca0 con 0x7fdd3c13f250 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c0a9120 con 0x7fdd3c0ae300 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c13e1c0 con 0x7fdd3c070a60 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.914+0000 7fdd40ada640 10 monclient(hunting): authenticate will time out at 3307.255627s 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd3ad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c13f250 0x7fdd3c1d8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd33fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd3c0ae300 0x7fdd3c13dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd33fff640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd3c0ae300 0x7fdd3c13dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:41262/0 (socket says 172.21.15.69:41262) 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd33fff640 1 -- 172.21.15.69:0/2603181941 learned_addr learned my addr 172.21.15.69:0/2603181941 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd33fff640 10 monclient(hunting): get_auth_request con 0x7fdd3c0ae300 auth_method 0 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd33fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd33fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd3ad9d640 10 monclient(hunting): get_auth_request con 0x7fdd3c13f250 auth_method 0 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd3ad9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.915+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.916+0000 7fdd3ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.916+0000 7fdd3ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.916+0000 7fdd3ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.916+0000 7fdd3a59c640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c070a60 0x7fdd3c142440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.916+0000 7fdd3a59c640 10 monclient(hunting): get_auth_request con 0x7fdd3c070a60 auth_method 0 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.916+0000 7fdd3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:48.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.916+0000 7fdd3a59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:48.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.916+0000 7fdd3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:48.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.917+0000 7fdd3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:48.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.917+0000 7fdd3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:48.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.917+0000 7fdd3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:48.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.918+0000 7fdd33fff640 10 monclient(hunting): handle_auth_done global_id 21562 payload 824 2024-01-21T05:05:48.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.918+0000 7fdd33fff640 1 -- 172.21.15.69:0/2603181941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c070a60 msgr2=0x7fdd3c142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.918+0000 7fdd33fff640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdd3c070a60 0x7fdd3c142440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.918+0000 7fdd33fff640 1 -- 172.21.15.69:0/2603181941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c13f250 msgr2=0x7fdd3c1d8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:48.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.918+0000 7fdd33fff640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3c13f250 0x7fdd3c1d8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:48.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.918+0000 7fdd33fff640 10 monclient: _finish_hunting 0 2024-01-21T05:05:48.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.918+0000 7fdd33fff640 1 monclient: found mon.noname-b 2024-01-21T05:05:48.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.918+0000 7fdd33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:48.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.918+0000 7fdd33fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:05:48.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.918+0000 7fdd33fff640 1 -- 172.21.15.69:0/2603181941 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd24009400 con 0x7fdd3c0ae300 2024-01-21T05:05:48.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.920+0000 7fdd33fff640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd3c0ae300 0x7fdd3c13dc80 secure :-1 s=READY pgs=5221 cs=0 l=1 rev1=1 crypto rx=0x7fdd240098f0 tx=0x7fdd240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:48.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.920+0000 7fdd32ffd640 10 client.?.objecter ms_handle_connect 0x7fdd3c0ae300 2024-01-21T05:05:48.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.920+0000 7fdd32ffd640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:48.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 1 -- 172.21.15.69:0/2603181941 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2403bc70 con 0x7fdd3c0ae300 2024-01-21T05:05:48.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:48.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 10 monclient: dump: 2024-01-21T05:05:48.926 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.926 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.926 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.926 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 10 monclient: _finish_auth 0 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 10 monclient: _check_auth_tickets 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd40ada640 5 monclient: authenticate success, global_id 21562 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 1 -- 172.21.15.69:0/2603181941 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2403be10 con 0x7fdd3c0ae300 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:48.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd40ada640 10 monclient: _renew_subs 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 1 -- 172.21.15.69:0/2603181941 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24045630 con 0x7fdd3c0ae300 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd40ada640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd40ada640 1 -- 172.21.15.69:0/2603181941 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c0a9320 con 0x7fdd3c0ae300 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd3959a640 4 set_mon_vals no callback set 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd32ffd640 10 monclient: dump: 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:48.928 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd40ada640 10 client.21562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd40ada640 10 monclient: _renew_subs 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd40ada640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd40ada640 1 -- 172.21.15.69:0/2603181941 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c13e3d0 con 0x7fdd3c0ae300 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.924+0000 7fdd40ada640 1 librados: init done 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.925+0000 7fdd40ada640 10 client.21562.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.925+0000 7fdd40ada640 15 client.21562.objecter _session_command_op_assign -1 1 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.925+0000 7fdd40ada640 20 client.21562.objecter _get_session osd=-1 returning homeless 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.925+0000 7fdd40ada640 10 client.21562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:48.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.925+0000 7fdd40ada640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:48.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.925+0000 7fdd40ada640 1 -- 172.21.15.69:0/2603181941 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdd3c0722b0 con 0x7fdd3c0ae300 2024-01-21T05:05:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.943+0000 7fdd32ffd640 1 -- 172.21.15.69:0/2603181941 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fdd2404e450 con 0x7fdd3c0ae300 2024-01-21T05:05:48.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.944+0000 7fdd32ffd640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdd1c06c8b0 0x7fdd1c06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.944+0000 7fdd3a59c640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdd1c06c8b0 0x7fdd1c06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.944+0000 7fdd3a59c640 10 monclient: get_auth_request con 0x7fdd1c06c8b0 auth_method 0 2024-01-21T05:05:48.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.945+0000 7fdd32ffd640 10 client.21562.objecter ms_handle_connect 0x7fdd1c06c8b0 2024-01-21T05:05:48.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.945+0000 7fdd3a59c640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdd1c06c8b0 0x7fdd1c06ed70 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0011e0 tx=0x7fdd2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:48.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 1 -- 172.21.15.69:0/2603181941 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1032..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7fdd2402e070 con 0x7fdd3c0ae300 2024-01-21T05:05:48.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 10 client.21562.objecter ms_dispatch 0x7fdd3c000b90 osd_map(1109..1109 src has 1032..1109) v4 2024-01-21T05:05:48.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 3 client.21562.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:48.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 3 client.21562.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:48.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 10 client.21562.objecter checking command 1 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x7fdd1c0745a0 0x7fdd1c076a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 20 client.21562.objecter _get_session s=0x7fdd1c073ed0 osd=1 3 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 20 client.21562.objecter put_session s=0x7fdd1c073ed0 osd=1 3 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 15 client.21562.objecter _session_command_op_remove -1 1 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 20 client.21562.objecter _get_session s=0x7fdd1c073ed0 osd=1 3 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 20 client.21562.objecter get_session s=0x7fdd1c073ed0 osd=1 3 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 15 client.21562.objecter _session_command_op_assign 1 1 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 20 client.21562.objecter put_session s=0x7fdd1c073ed0 osd=1 4 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 10 client.21562.objecter _send_command 1 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 1 -- 172.21.15.69:0/2603181941 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd1c077110 con 0x7fdd1c0745a0 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 20 client.21562.objecter dump_active .. 0 homeless 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 1 -- 172.21.15.69:0/2603181941 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=1109) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd24076d40 con 0x7fdd3c0ae300 2024-01-21T05:05:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.946+0000 7fdd32ffd640 10 monclient: handle_get_version_reply finishing 1 version 1109 2024-01-21T05:05:48.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.947+0000 7fdd3ad9d640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x7fdd1c0745a0 0x7fdd1c076a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:48.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.947+0000 7fdd3ad9d640 10 monclient: get_auth_request con 0x7fdd1c0745a0 auth_method 0 2024-01-21T05:05:48.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.950+0000 7fdd3ad9d640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x7fdd1c0745a0 0x7fdd1c076a40 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:48.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.950+0000 7fdd32ffd640 10 client.21562.objecter ms_handle_connect 0x7fdd1c0745a0 2024-01-21T05:05:48.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.975+0000 7fdd32ffd640 1 -- 172.21.15.69:0/2603181941 <== osd.1 v2:172.21.15.69:6808/3868480563 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdd1c077110 con 0x7fdd1c0745a0 2024-01-21T05:05:48.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.975+0000 7fdd32ffd640 10 client.21562.objecter ms_dispatch 0x7fdd3c000b90 command_reply(tid 1: 0 ) v1 2024-01-21T05:05:48.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.975+0000 7fdd32ffd640 10 client.21562.objecter _finish_command 1 = system:0 2024-01-21T05:05:48.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.975+0000 7fdd32ffd640 20 client.21562.objecter put_session s=0x7fdd1c073ed0 osd=1 4 2024-01-21T05:05:48.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.975+0000 7fdd32ffd640 15 client.21562.objecter _session_command_op_remove 1 1 2024-01-21T05:05:48.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.989+0000 7fdd40ada640 10 client.21562.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T05:05:48.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.989+0000 7fdd40ada640 15 client.21562.objecter _session_command_op_assign -1 2 2024-01-21T05:05:48.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.989+0000 7fdd40ada640 20 client.21562.objecter _get_session s=0x7fdd1c073ed0 osd=1 3 2024-01-21T05:05:48.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.989+0000 7fdd40ada640 20 client.21562.objecter put_session s=0x7fdd1c073ed0 osd=1 3 2024-01-21T05:05:48.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.989+0000 7fdd40ada640 20 client.21562.objecter _get_session s=0x7fdd1c073ed0 osd=1 3 2024-01-21T05:05:48.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.989+0000 7fdd40ada640 15 client.21562.objecter _session_command_op_remove -1 2 2024-01-21T05:05:48.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.989+0000 7fdd40ada640 20 client.21562.objecter get_session s=0x7fdd1c073ed0 osd=1 3 2024-01-21T05:05:48.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.989+0000 7fdd40ada640 15 client.21562.objecter _session_command_op_assign 1 2 2024-01-21T05:05:48.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.989+0000 7fdd40ada640 20 client.21562.objecter put_session s=0x7fdd1c073ed0 osd=1 4 2024-01-21T05:05:48.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.989+0000 7fdd40ada640 10 client.21562.objecter _send_command 2 2024-01-21T05:05:48.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:48.989+0000 7fdd40ada640 1 -- 172.21.15.69:0/2603181941 --> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdd3c04e970 con 0x7fdd1c0745a0 2024-01-21T05:05:49.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd32ffd640 1 -- 172.21.15.69:0/2603181941 <== osd.1 v2:172.21.15.69:6808/3868480563 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fdd3c04e970 con 0x7fdd1c0745a0 2024-01-21T05:05:49.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd32ffd640 10 client.21562.objecter ms_dispatch 0x7fdd3c000b90 command_reply(tid 2: 0 ) v1 2024-01-21T05:05:49.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd32ffd640 10 client.21562.objecter _finish_command 2 = system:0 2024-01-21T05:05:49.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd32ffd640 20 client.21562.objecter put_session s=0x7fdd1c073ed0 osd=1 4 2024-01-21T05:05:49.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd32ffd640 15 client.21562.objecter _session_command_op_remove 1 2 2024-01-21T05:05:49.004 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-21T05:05:49.004 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T05:05:49.004 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-21T05:05:49.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 10 librados: watch_flush enter 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 10 librados: watch_flush exit 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 10 client.21562.objecter close_session for osd.1 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 1 -- 172.21.15.69:0/2603181941 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x7fdd1c0745a0 msgr2=0x7fdd1c076a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.69:6808/3868480563,v1:172.21.15.69:6809/3868480563] conn(0x7fdd1c0745a0 0x7fdd1c076a40 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 20 client.21562.objecter put_session s=0x7fdd1c073ed0 osd=1 1 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 20 client.21562.objecter shutdown clearing up homeless session... 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 10 client.21562.objecter successfully canceled tick 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 1 -- 172.21.15.69:0/2603181941 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdd1c06c8b0 msgr2=0x7fdd1c06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdd1c06c8b0 0x7fdd1c06ed70 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0011e0 tx=0x7fdd2c009040 comp rx=0 tx=0).stop 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 10 monclient: shutdown 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 1 -- 172.21.15.69:0/2603181941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd3c0ae300 msgr2=0x7fdd3c13dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 1 --2- 172.21.15.69:0/2603181941 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdd3c0ae300 0x7fdd3c13dc80 secure :-1 s=READY pgs=5221 cs=0 l=1 rev1=1 crypto rx=0x7fdd240098f0 tx=0x7fdd240046c0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd3ad9d640 1 -- 172.21.15.69:0/2603181941 reap_dead start 2024-01-21T05:05:49.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 1 -- 172.21.15.69:0/2603181941 shutdown_connections 2024-01-21T05:05:49.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 1 -- 172.21.15.69:0/2603181941 >> 172.21.15.69:0/2603181941 conn(0x7fdd3c0b3690 msgr2=0x7fdd3c0b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:49.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 1 -- 172.21.15.69:0/2603181941 shutdown_connections 2024-01-21T05:05:49.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 1 -- 172.21.15.69:0/2603181941 wait complete. 2024-01-21T05:05:49.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.003+0000 7fdd40ada640 1 librados: shutdown 2024-01-21T05:05:49.015 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-21T05:05:49.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.116+0000 7f4b0b2a2640 10 monclient: get_monmap_and_config 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.116+0000 7f4b0b2a2640 10 monclient: build_initial_monmap 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.117+0000 7f4b0b2a2640 10 monclient: monmap: 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:49.117993+0000 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:49.117993+0000 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:49.118 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:49.119 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.117+0000 7f4b0b2a2640 1 Processor -- start 2024-01-21T05:05:49.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.117+0000 7f4b0b2a2640 1 -- start start 2024-01-21T05:05:49.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.117+0000 7f4b0b2a2640 10 monclient: init 2024-01-21T05:05:49.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:49.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-01-21T05:05:49.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b04066620 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:49.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b04071d80 0x7f4b0413f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 10 monclient(hunting): picked mon.noname-b con 0x7f4b04071d80 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b040afc50 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b040abb40 con 0x7f4b04066620 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b040ad280 con 0x7f4b040afc50 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b04066a00 con 0x7f4b04071d80 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.118+0000 7f4b0b2a2640 10 monclient(hunting): authenticate will time out at 3307.459683s 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.119+0000 7f4b0883d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b04071d80 0x7f4b0413f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.119+0000 7f4b0903e640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4b04066620 0x7f4b0413f240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50216/0 (socket says 172.21.15.69:50216) 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.119+0000 7f4b0903e640 1 -- 172.21.15.69:0/158040171 learned_addr learned my addr 172.21.15.69:0/158040171 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.119+0000 7f4b0883d640 10 monclient(hunting): get_auth_request con 0x7f4b04071d80 auth_method 0 2024-01-21T05:05:49.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.119+0000 7f4b0883d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:49.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.119+0000 7f4b0883d640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:49.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.119+0000 7f4b0883d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:49.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.119+0000 7f4b0883d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:49.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.119+0000 7f4b0883d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:49.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.119+0000 7f4b0883d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:49.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 1 -- 172.21.15.69:0/158040171 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514979684 0 0) 0x7f4b040ad280 con 0x7f4b040afc50 2024-01-21T05:05:49.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 10 monclient(hunting): _negotiate 2024-01-21T05:05:49.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:49.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:49.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 10 monclient(hunting): my global_id is 21565 2024-01-21T05:05:49.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 1 -- 172.21.15.69:0/158040171 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af0003520 con 0x7f4b040afc50 2024-01-21T05:05:49.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 1 -- 172.21.15.69:0/158040171 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923839407 0 0) 0x7f4b040abb40 con 0x7f4b04066620 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 10 monclient(hunting): _negotiate 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 10 monclient(hunting): my global_id is 17454 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4afbfff640 1 -- 172.21.15.69:0/158040171 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b040ad280 con 0x7f4b04066620 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4b0883d640 10 monclient(hunting): handle_auth_done global_id 18833 payload 274 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4b0883d640 1 -- 172.21.15.69:0/158040171 >> v1:172.21.15.144:6789/0 conn(0x7f4b040afc50 legacy=0x7f4b040b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4b0883d640 1 -- 172.21.15.69:0/158040171 >> v1:172.21.15.69:6789/0 conn(0x7f4b04066620 legacy=0x7f4b0413f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4b0883d640 10 monclient: _finish_hunting 0 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4b0883d640 1 monclient: found mon.noname-b 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4b0883d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4b0883d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.121+0000 7f4b0883d640 1 -- 172.21.15.69:0/158040171 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b040b07f0 con 0x7f4b04071d80 2024-01-21T05:05:49.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.123+0000 7f4b0883d640 1 --2- 172.21.15.69:0/158040171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b04071d80 0x7f4b0413f950 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7f4af40097d0 tx=0x7f4af402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4d8c14bd194dcba server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:49.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 1 -- 172.21.15.69:0/158040171 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4031890 con 0x7f4b04071d80 2024-01-21T05:05:49.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:49.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:49.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 10 monclient: dump: 2024-01-21T05:05:49.129 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 10 monclient: _finish_auth 0 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 10 monclient: _check_auth_tickets 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 1 -- 172.21.15.69:0/158040171 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af40319f0 con 0x7f4b04071d80 2024-01-21T05:05:49.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4b0b2a2640 5 monclient: authenticate success, global_id 18833 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4b0b2a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4b0b2a2640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4b0b2a2640 4 set_mon_vals no callback set 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 1 -- 172.21.15.69:0/158040171 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4031d80 con 0x7f4b04071d80 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.127+0000 7f4afbfff640 10 monclient: dump: 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:49.131 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.132 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:49.132 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:49.132 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:49.132 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.128+0000 7f4b0b2a2640 10 monclient: shutdown 2024-01-21T05:05:49.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.128+0000 7f4b0b2a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:49.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.128+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/158040171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b04071d80 msgr2=0x7f4b0413f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.128+0000 7f4b0b2a2640 1 --2- 172.21.15.69:0/158040171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b04071d80 0x7f4b0413f950 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7f4af40097d0 tx=0x7f4af402b930 comp rx=0 tx=0).stop 2024-01-21T05:05:49.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.128+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/158040171 shutdown_connections 2024-01-21T05:05:49.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.128+0000 7f4b0b2a2640 1 --2- 172.21.15.69:0/158040171 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b04071d80 0x7f4b0413f950 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.128+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/158040171 >> 172.21.15.69:0/158040171 conn(0x7f4b0406a630 msgr2=0x7f4b0406aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:49.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.128+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/158040171 shutdown_connections 2024-01-21T05:05:49.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.128+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/158040171 wait complete. 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.129+0000 7f4b0b2a2640 10 monclient: build_initial_monmap 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.129+0000 7f4b0b2a2640 10 monclient: monmap: 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:49.136 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.129+0000 7f4b0b2a2640 1 librados: starting msgr at 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.129+0000 7f4b0b2a2640 1 librados: starting objecter 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.129+0000 7f4b0b2a2640 1 Processor -- start 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.129+0000 7f4b0b2a2640 1 -- start start 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.129+0000 7f4b0b2a2640 1 librados: setting wanted keys 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.129+0000 7f4b0b2a2640 1 librados: calling monclient init 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.129+0000 7f4b0b2a2640 10 monclient: init 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b04066620 0x7f4b041429c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 10 monclient(hunting): picked mon.noname-a con 0x7f4b04066620 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b04071d80 0x7f4b04141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 10 monclient(hunting): picked mon.noname-b con 0x7f4b04071d80 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:49.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b040afc50 0x7f4b041365e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 10 monclient(hunting): picked mon.noname-c con 0x7f4b040afc50 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b04143ee0 con 0x7f4b04066620 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b041da4f0 con 0x7f4b04071d80 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b04144130 con 0x7f4b040afc50 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0b2a2640 10 monclient(hunting): authenticate will time out at 3307.471666s 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0903e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b04066620 0x7f4b041429c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.130+0000 7f4b0983f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b040afc50 0x7f4b041365e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0883d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b04071d80 0x7f4b04141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:49.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0903e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b04066620 0x7f4b041429c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41630/0 (socket says 172.21.15.69:41630) 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0983f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b040afc50 0x7f4b041365e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:48582/0 (socket says 172.21.15.69:48582) 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0903e640 1 -- 172.21.15.69:0/3232701537 learned_addr learned my addr 172.21.15.69:0/3232701537 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0903e640 10 monclient(hunting): get_auth_request con 0x7f4b04066620 auth_method 0 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0903e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0903e640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0903e640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0983f640 10 monclient(hunting): get_auth_request con 0x7f4b040afc50 auth_method 0 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0983f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0983f640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0983f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0883d640 10 monclient(hunting): get_auth_request con 0x7f4b04071d80 auth_method 0 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0883d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:49.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0883d640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0883d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0983f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0983f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0983f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0903e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0903e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.131+0000 7f4b0903e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.132+0000 7f4b0883d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.132+0000 7f4b0883d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.132+0000 7f4b0883d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.134+0000 7f4b0883d640 10 monclient(hunting): handle_auth_done global_id 21568 payload 824 2024-01-21T05:05:49.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.134+0000 7f4b0883d640 1 -- 172.21.15.69:0/3232701537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b040afc50 msgr2=0x7f4b041365e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.134+0000 7f4b0883d640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b040afc50 0x7f4b041365e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.134+0000 7f4b0883d640 1 -- 172.21.15.69:0/3232701537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b04066620 msgr2=0x7f4b041429c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.134+0000 7f4b0883d640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b04066620 0x7f4b041429c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.134+0000 7f4b0883d640 10 monclient: _finish_hunting 0 2024-01-21T05:05:49.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.134+0000 7f4b0883d640 1 monclient: found mon.noname-b 2024-01-21T05:05:49.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.134+0000 7f4b0883d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:49.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.134+0000 7f4b0883d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:05:49.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.134+0000 7f4b0883d640 1 -- 172.21.15.69:0/3232701537 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af4009400 con 0x7f4b04071d80 2024-01-21T05:05:49.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.134+0000 7f4b0903e640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b04066620 0x7f4b041429c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T05:05:49.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.136+0000 7f4b0883d640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b04071d80 0x7f4b04141d40 secure :-1 s=READY pgs=5223 cs=0 l=1 rev1=1 crypto rx=0x7f4af4033000 tx=0x7f4af40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:49.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.136+0000 7f4af9ffb640 10 client.?.objecter ms_handle_connect 0x7f4b04071d80 2024-01-21T05:05:49.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.137+0000 7f4af9ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:49.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 1 -- 172.21.15.69:0/3232701537 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af403bc70 con 0x7f4b04071d80 2024-01-21T05:05:49.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:49.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 10 monclient: dump: 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 10 monclient: _finish_auth 0 2024-01-21T05:05:49.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 10 monclient: _check_auth_tickets 2024-01-21T05:05:49.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:49.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 1 -- 172.21.15.69:0/3232701537 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af403be10 con 0x7f4b04071d80 2024-01-21T05:05:49.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:49.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 1 -- 172.21.15.69:0/3232701537 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4045630 con 0x7f4b04071d80 2024-01-21T05:05:49.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:49.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:49.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4af9ffb640 10 monclient: dump: 2024-01-21T05:05:49.145 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:49.145 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:49.145 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.145 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4b0b2a2640 5 monclient: authenticate success, global_id 21568 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4b0b2a2640 10 monclient: _renew_subs 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4afb7fe640 4 set_mon_vals no callback set 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4b0b2a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/3232701537 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b04142280 con 0x7f4b04071d80 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4b0b2a2640 10 client.21568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4b0b2a2640 10 monclient: _renew_subs 2024-01-21T05:05:49.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4b0b2a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:49.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.140+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/3232701537 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b04136b20 con 0x7f4b04071d80 2024-01-21T05:05:49.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.141+0000 7f4b0b2a2640 1 librados: init done 2024-01-21T05:05:49.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.141+0000 7f4b0b2a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:49.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.141+0000 7f4b0b2a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:49.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.141+0000 7f4b0b2a2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:49.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.141+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/3232701537 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b04142f00 con 0x7f4b04071d80 2024-01-21T05:05:49.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.388+0000 7f4af9ffb640 1 -- 172.21.15.69:0/3232701537 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f4af404e430 con 0x7f4b04071d80 2024-01-21T05:05:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.388+0000 7f4af9ffb640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4ae806c8e0 0x7f4ae806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.388+0000 7f4b0903e640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4ae806c8e0 0x7f4ae806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.388+0000 7f4b0903e640 10 monclient: get_auth_request con 0x7f4ae806c8e0 auth_method 0 2024-01-21T05:05:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.389+0000 7f4af9ffb640 10 client.21568.objecter ms_handle_connect 0x7f4ae806c8e0 2024-01-21T05:05:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.389+0000 7f4b0903e640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4ae806c8e0 0x7f4ae806eda0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0011e0 tx=0x7f4afc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.391+0000 7f4af9ffb640 1 -- 172.21.15.69:0/3232701537 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1032..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f4af407a970 con 0x7f4b04071d80 2024-01-21T05:05:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.391+0000 7f4af9ffb640 10 client.21568.objecter ms_dispatch 0x7f4b04000b90 osd_map(1109..1109 src has 1032..1109) v4 2024-01-21T05:05:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.391+0000 7f4af9ffb640 3 client.21568.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.391+0000 7f4af9ffb640 3 client.21568.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.391+0000 7f4af9ffb640 20 client.21568.objecter dump_active .. 0 homeless 2024-01-21T05:05:49.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.401+0000 7f4af9ffb640 1 -- 172.21.15.69:0/3232701537 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f4af4076cb0 con 0x7f4b04071d80 2024-01-21T05:05:49.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.401+0000 7f4af9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:49.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.401+0000 7f4af9ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:49.526 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:17.433 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:49.530 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:17.437 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:49.542 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:17.449 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:49.579 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:17.454 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:49.580 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:17.455 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:49.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.589+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/3232701537 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b04142f00 con 0x7f4ae806c8e0 2024-01-21T05:05:49.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.589+0000 7f4af9ffb640 1 -- 172.21.15.69:0/3232701537 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4b04142f00 con 0x7f4ae806c8e0 2024-01-21T05:05:49.590 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 1 to deep-scrub 2024-01-21T05:05:49.592 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:17.467 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:49.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 10 librados: watch_flush enter 2024-01-21T05:05:49.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 10 librados: watch_flush exit 2024-01-21T05:05:49.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 20 client.21568.objecter shutdown clearing up homeless session... 2024-01-21T05:05:49.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 10 client.21568.objecter successfully canceled tick 2024-01-21T05:05:49.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/3232701537 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4ae806c8e0 msgr2=0x7f4ae806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4ae806c8e0 0x7f4ae806eda0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f4afc0011e0 tx=0x7f4afc009040 comp rx=0 tx=0).stop 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 10 monclient: shutdown 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/3232701537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b04071d80 msgr2=0x7f4b04141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b04071d80 0x7f4b04141d40 secure :-1 s=READY pgs=5223 cs=0 l=1 rev1=1 crypto rx=0x7f4af4033000 tx=0x7f4af40046c0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/3232701537 shutdown_connections 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f4ae806c8e0 0x7f4ae806eda0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f4b040afc50 0x7f4b041365e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b04071d80 0x7f4b04141d40 unknown :-1 s=CLOSED pgs=5223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 --2- 172.21.15.69:0/3232701537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b04066620 0x7f4b041429c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/3232701537 >> 172.21.15.69:0/3232701537 conn(0x7f4b0406a630 msgr2=0x7f4b0413d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/3232701537 shutdown_connections 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 -- 172.21.15.69:0/3232701537 wait complete. 2024-01-21T05:05:49.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.592+0000 7f4b0b2a2640 1 librados: shutdown 2024-01-21T05:05:49.603 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:17.510 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:49.604 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:17.511 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:49.605 INFO:tasks.ceph:Scrubbing osd.2 2024-01-21T05:05:49.605 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-21T05:05:49.617 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:17.524 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.713+0000 7fdb67fd0640 10 monclient: get_monmap_and_config 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.713+0000 7fdb67fd0640 10 monclient: build_initial_monmap 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.713+0000 7fdb67fd0640 10 monclient: monmap: 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:49.714657+0000 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:49.714657+0000 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.714+0000 7fdb67fd0640 1 Processor -- start 2024-01-21T05:05:49.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.714+0000 7fdb67fd0640 1 -- start start 2024-01-21T05:05:49.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.714+0000 7fdb67fd0640 10 monclient: init 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdb600a8a40 0x7fdb60145b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 10 monclient(hunting): picked mon.noname-a con 0x7fdb600a8a40 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb600b2210 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdb600a8e20 0x7fdb6014c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 10 monclient(hunting): picked mon.noname-c con 0x7fdb600a8e20 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb60145700 con 0x7fdb600b2210 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb600aa7b0 con 0x7fdb600a8a40 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb600aa9b0 con 0x7fdb600a8e20 2024-01-21T05:05:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:49.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.715+0000 7fdb67fd0640 10 monclient(hunting): authenticate will time out at 3308.056316s 2024-01-21T05:05:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.724+0000 7fdb65d6c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdb600a8a40 0x7fdb60145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.729+0000 7fdb65d6c640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdb600a8a40 0x7fdb60145b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41642/0 (socket says 172.21.15.69:41642) 2024-01-21T05:05:49.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.729+0000 7fdb65d6c640 1 -- 172.21.15.69:0/4190385080 learned_addr learned my addr 172.21.15.69:0/4190385080 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:49.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.729+0000 7fdb65d6c640 10 monclient(hunting): get_auth_request con 0x7fdb600a8a40 auth_method 0 2024-01-21T05:05:49.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.729+0000 7fdb65d6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:49.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.729+0000 7fdb65d6c640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:49.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.729+0000 7fdb65d6c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:49.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.735+0000 7fdb65d6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:49.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.735+0000 7fdb65d6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:49.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.735+0000 7fdb65d6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:49.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.735+0000 7fdb6656d640 1 --2- 172.21.15.69:0/4190385080 >> v2:172.21.15.144:3300/0 conn(0x7fdb600a8e20 0x7fdb6014c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.744+0000 7fdb65d6c640 10 monclient(hunting): handle_auth_done global_id 17460 payload 274 2024-01-21T05:05:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.744+0000 7fdb65d6c640 1 -- 172.21.15.69:0/4190385080 >> v2:172.21.15.144:3300/0 conn(0x7fdb600a8e20 msgr2=0x7fdb6014c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.744+0000 7fdb65d6c640 1 --2- 172.21.15.69:0/4190385080 >> v2:172.21.15.144:3300/0 conn(0x7fdb600a8e20 0x7fdb6014c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.744+0000 7fdb65d6c640 1 -- 172.21.15.69:0/4190385080 >> v1:172.21.15.69:6789/0 conn(0x7fdb600b2210 legacy=0x7fdb6013f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.744+0000 7fdb65d6c640 10 monclient: _finish_hunting 0 2024-01-21T05:05:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.744+0000 7fdb65d6c640 1 monclient: found mon.noname-a 2024-01-21T05:05:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.744+0000 7fdb65d6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:49.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.744+0000 7fdb65d6c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:49.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.744+0000 7fdb65d6c640 1 -- 172.21.15.69:0/4190385080 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb600aac90 con 0x7fdb600a8a40 2024-01-21T05:05:49.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.762+0000 7fdb65d6c640 1 --2- 172.21.15.69:0/4190385080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb600a8a40 0x7fdb60145b90 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7fdb60145670 tx=0x7fdb5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5be8f3faca0786 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:49.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 1 -- 172.21.15.69:0/4190385080 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c031870 con 0x7fdb600a8a40 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 10 monclient: dump: 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:49.786 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 10 monclient: _finish_auth 0 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 10 monclient: _check_auth_tickets 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 1 -- 172.21.15.69:0/4190385080 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5c0319d0 con 0x7fdb600a8a40 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 5 monclient: authenticate success, global_id 17460 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 4 set_mon_vals no callback set 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 1 -- 172.21.15.69:0/4190385080 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c031d60 con 0x7fdb600a8a40 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb64d6a640 10 monclient: dump: 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:49.788 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.788 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.788 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:49.788 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.788 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:49.788 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:49.788 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:49.788 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 10 monclient: shutdown 2024-01-21T05:05:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 1 -- 172.21.15.69:0/4190385080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb600a8a40 msgr2=0x7fdb60145b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 1 --2- 172.21.15.69:0/4190385080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb600a8a40 0x7fdb60145b90 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7fdb60145670 tx=0x7fdb5c02b910 comp rx=0 tx=0).stop 2024-01-21T05:05:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 1 -- 172.21.15.69:0/4190385080 shutdown_connections 2024-01-21T05:05:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 1 --2- 172.21.15.69:0/4190385080 >> v2:172.21.15.144:3300/0 conn(0x7fdb600a8e20 0x7fdb6014c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 1 --2- 172.21.15.69:0/4190385080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb600a8a40 0x7fdb60145b90 unknown :-1 s=CLOSED pgs=3495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 1 -- 172.21.15.69:0/4190385080 >> 172.21.15.69:0/4190385080 conn(0x7fdb60066f10 msgr2=0x7fdb600b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 1 -- 172.21.15.69:0/4190385080 shutdown_connections 2024-01-21T05:05:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.785+0000 7fdb67fd0640 1 -- 172.21.15.69:0/4190385080 wait complete. 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.786+0000 7fdb67fd0640 10 monclient: build_initial_monmap 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.786+0000 7fdb67fd0640 10 monclient: monmap: 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.786+0000 7fdb67fd0640 1 librados: starting msgr at 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.786+0000 7fdb67fd0640 1 librados: starting objecter 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.786+0000 7fdb67fd0640 1 Processor -- start 2024-01-21T05:05:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.786+0000 7fdb67fd0640 1 -- start start 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.786+0000 7fdb67fd0640 1 librados: setting wanted keys 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.786+0000 7fdb67fd0640 1 librados: calling monclient init 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.786+0000 7fdb67fd0640 10 monclient: init 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb600a8a40 0x7fdb600ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 10 monclient(hunting): picked mon.noname-a con 0x7fdb600a8a40 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb600a8e20 0x7fdb60140300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 10 monclient(hunting): picked mon.noname-c con 0x7fdb600a8e20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb600b2210 0x7fdb60140840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 10 monclient(hunting): picked mon.noname-b con 0x7fdb600b2210 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb600aced0 con 0x7fdb600a8a40 2024-01-21T05:05:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:49.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb600ad0d0 con 0x7fdb600b2210 2024-01-21T05:05:49.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:49.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb600ad3b0 con 0x7fdb600a8e20 2024-01-21T05:05:49.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:49.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.787+0000 7fdb67fd0640 10 monclient(hunting): authenticate will time out at 3308.128719s 2024-01-21T05:05:49.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.791+0000 7fdb65d6c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb600a8a40 0x7fdb600ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:49.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.792+0000 7fdb6556b640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb600a8e20 0x7fdb60140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:49.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.793+0000 7fdb65d6c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb600a8a40 0x7fdb600ab6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41654/0 (socket says 172.21.15.69:41654) 2024-01-21T05:05:49.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.793+0000 7fdb65d6c640 1 -- 172.21.15.69:0/3113459401 learned_addr learned my addr 172.21.15.69:0/3113459401 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:49.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.793+0000 7fdb65d6c640 10 monclient(hunting): get_auth_request con 0x7fdb600a8a40 auth_method 0 2024-01-21T05:05:49.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.793+0000 7fdb65d6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:49.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.793+0000 7fdb65d6c640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:49.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.793+0000 7fdb65d6c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:49.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.794+0000 7fdb6556b640 10 monclient(hunting): get_auth_request con 0x7fdb600a8e20 auth_method 0 2024-01-21T05:05:49.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.794+0000 7fdb6556b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:49.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.794+0000 7fdb6556b640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:49.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.794+0000 7fdb6556b640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.796+0000 7fdb65d6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.796+0000 7fdb65d6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.796+0000 7fdb65d6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:49.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.800+0000 7fdb65d6c640 10 monclient(hunting): handle_auth_done global_id 17463 payload 824 2024-01-21T05:05:49.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.800+0000 7fdb65d6c640 1 -- 172.21.15.69:0/3113459401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb600a8e20 msgr2=0x7fdb60140300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.800+0000 7fdb65d6c640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb600a8e20 0x7fdb60140300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.800+0000 7fdb65d6c640 1 -- 172.21.15.69:0/3113459401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb600b2210 msgr2=0x7fdb60140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:49.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.800+0000 7fdb65d6c640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb600b2210 0x7fdb60140840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:49.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.800+0000 7fdb65d6c640 10 monclient: _finish_hunting 0 2024-01-21T05:05:49.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.800+0000 7fdb65d6c640 1 monclient: found mon.noname-a 2024-01-21T05:05:49.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.800+0000 7fdb65d6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:49.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.800+0000 7fdb65d6c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:49.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.800+0000 7fdb65d6c640 1 -- 172.21.15.69:0/3113459401 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0094d0 con 0x7fdb600a8a40 2024-01-21T05:05:49.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.810+0000 7fdb65d6c640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb600a8a40 0x7fdb600ab6e0 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c000c00 tx=0x7fdb5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:49.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.810+0000 7fdb567fc640 10 client.?.objecter ms_handle_connect 0x7fdb600a8a40 2024-01-21T05:05:49.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.810+0000 7fdb567fc640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:49.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.879+0000 7fdb567fc640 1 -- 172.21.15.69:0/3113459401 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c03adf0 con 0x7fdb600a8a40 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.879+0000 7fdb567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.879+0000 7fdb567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.879+0000 7fdb567fc640 10 monclient: dump: 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:49.881 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.879+0000 7fdb567fc640 10 monclient: _finish_auth 0 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.879+0000 7fdb567fc640 10 monclient: _check_auth_tickets 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.879+0000 7fdb567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.879+0000 7fdb567fc640 1 -- 172.21.15.69:0/3113459401 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5c0443f0 con 0x7fdb600a8a40 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.879+0000 7fdb567fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.879+0000 7fdb567fc640 1 -- 172.21.15.69:0/3113459401 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5c0447a0 con 0x7fdb600a8a40 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb567fc640 10 monclient: dump: 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:49.882 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 5 monclient: authenticate success, global_id 17463 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 10 monclient: _renew_subs 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb57fff640 4 set_mon_vals no callback set 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 1 -- 172.21.15.69:0/3113459401 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb600ad690 con 0x7fdb600a8a40 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 10 client.17463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 10 monclient: _renew_subs 2024-01-21T05:05:49.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:49.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 1 -- 172.21.15.69:0/3113459401 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb600ac200 con 0x7fdb600a8a40 2024-01-21T05:05:49.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 1 librados: init done 2024-01-21T05:05:49.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 10 client.17463.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:49.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 15 client.17463.objecter _session_command_op_assign -1 1 2024-01-21T05:05:49.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 20 client.17463.objecter _get_session osd=-1 returning homeless 2024-01-21T05:05:49.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 10 client.17463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:49.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:49.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.880+0000 7fdb67fd0640 1 -- 172.21.15.69:0/3113459401 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb600aa1c0 con 0x7fdb600a8a40 2024-01-21T05:05:49.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.941+0000 7fdb567fc640 1 -- 172.21.15.69:0/3113459401 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fdb5c04d450 con 0x7fdb600a8a40 2024-01-21T05:05:49.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.942+0000 7fdb567fc640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdb4406c890 0x7fdb4406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.942+0000 7fdb6556b640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdb4406c890 0x7fdb4406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:49.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.942+0000 7fdb6556b640 10 monclient: get_auth_request con 0x7fdb4406c890 auth_method 0 2024-01-21T05:05:49.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.943+0000 7fdb6556b640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdb4406c890 0x7fdb4406ed50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdb50006fd0 tx=0x7fdb50008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:49.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.943+0000 7fdb567fc640 10 client.17463.objecter ms_handle_connect 0x7fdb4406c890 2024-01-21T05:05:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.947+0000 7fdb567fc640 1 -- 172.21.15.69:0/3113459401 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7fdb5c02d040 con 0x7fdb600a8a40 2024-01-21T05:05:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.947+0000 7fdb567fc640 10 client.17463.objecter ms_dispatch 0x7fdb60000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.947+0000 7fdb567fc640 3 client.17463.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.947+0000 7fdb567fc640 3 client.17463.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 10 client.17463.objecter checking command 1 2024-01-21T05:05:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] conn(0x7fdb44074580 0x7fdb44076a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 20 client.17463.objecter _get_session s=0x7fdb44073eb0 osd=2 3 2024-01-21T05:05:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 20 client.17463.objecter put_session s=0x7fdb44073eb0 osd=2 3 2024-01-21T05:05:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 15 client.17463.objecter _session_command_op_remove -1 1 2024-01-21T05:05:49.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 20 client.17463.objecter _get_session s=0x7fdb44073eb0 osd=2 3 2024-01-21T05:05:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 20 client.17463.objecter get_session s=0x7fdb44073eb0 osd=2 3 2024-01-21T05:05:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 15 client.17463.objecter _session_command_op_assign 2 1 2024-01-21T05:05:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 20 client.17463.objecter put_session s=0x7fdb44073eb0 osd=2 4 2024-01-21T05:05:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 10 client.17463.objecter _send_command 1 2024-01-21T05:05:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 1 -- 172.21.15.69:0/3113459401 --> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb440770f0 con 0x7fdb44074580 2024-01-21T05:05:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 20 client.17463.objecter dump_active .. 0 homeless 2024-01-21T05:05:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 1 -- 172.21.15.69:0/3113459401 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=1109) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb5c0771e0 con 0x7fdb600a8a40 2024-01-21T05:05:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb567fc640 10 monclient: handle_get_version_reply finishing 1 version 1109 2024-01-21T05:05:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.948+0000 7fdb6656d640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] conn(0x7fdb44074580 0x7fdb44076a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:49.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.949+0000 7fdb6656d640 10 monclient: get_auth_request con 0x7fdb44074580 auth_method 0 2024-01-21T05:05:49.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.951+0000 7fdb567fc640 10 client.17463.objecter ms_handle_connect 0x7fdb44074580 2024-01-21T05:05:49.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.951+0000 7fdb6656d640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] conn(0x7fdb44074580 0x7fdb44076a20 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.977+0000 7fdb567fc640 1 -- 172.21.15.69:0/3113459401 <== osd.2 v2:172.21.15.69:6816/3954012353 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdb440770f0 con 0x7fdb44074580 2024-01-21T05:05:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.977+0000 7fdb567fc640 10 client.17463.objecter ms_dispatch 0x7fdb60000b90 command_reply(tid 1: 0 ) v1 2024-01-21T05:05:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.977+0000 7fdb567fc640 10 client.17463.objecter _finish_command 1 = system:0 2024-01-21T05:05:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.977+0000 7fdb567fc640 20 client.17463.objecter put_session s=0x7fdb44073eb0 osd=2 4 2024-01-21T05:05:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.977+0000 7fdb567fc640 15 client.17463.objecter _session_command_op_remove 2 1 2024-01-21T05:05:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.992+0000 7fdb67fd0640 10 client.17463.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T05:05:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.992+0000 7fdb67fd0640 15 client.17463.objecter _session_command_op_assign -1 2 2024-01-21T05:05:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.992+0000 7fdb67fd0640 20 client.17463.objecter _get_session s=0x7fdb44073eb0 osd=2 3 2024-01-21T05:05:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.992+0000 7fdb67fd0640 20 client.17463.objecter put_session s=0x7fdb44073eb0 osd=2 3 2024-01-21T05:05:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.992+0000 7fdb67fd0640 20 client.17463.objecter _get_session s=0x7fdb44073eb0 osd=2 3 2024-01-21T05:05:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.992+0000 7fdb67fd0640 15 client.17463.objecter _session_command_op_remove -1 2 2024-01-21T05:05:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.992+0000 7fdb67fd0640 20 client.17463.objecter get_session s=0x7fdb44073eb0 osd=2 3 2024-01-21T05:05:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.992+0000 7fdb67fd0640 15 client.17463.objecter _session_command_op_assign 2 2 2024-01-21T05:05:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.992+0000 7fdb67fd0640 20 client.17463.objecter put_session s=0x7fdb44073eb0 osd=2 4 2024-01-21T05:05:49.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.992+0000 7fdb67fd0640 10 client.17463.objecter _send_command 2 2024-01-21T05:05:49.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:49.992+0000 7fdb67fd0640 1 -- 172.21.15.69:0/3113459401 --> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdb6004e970 con 0x7fdb44074580 2024-01-21T05:05:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.008+0000 7fdb567fc640 1 -- 172.21.15.69:0/3113459401 <== osd.2 v2:172.21.15.69:6816/3954012353 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fdb6004e970 con 0x7fdb44074580 2024-01-21T05:05:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.008+0000 7fdb567fc640 10 client.17463.objecter ms_dispatch 0x7fdb60000b90 command_reply(tid 2: 0 ) v1 2024-01-21T05:05:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.008+0000 7fdb567fc640 10 client.17463.objecter _finish_command 2 = system:0 2024-01-21T05:05:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.008+0000 7fdb567fc640 20 client.17463.objecter put_session s=0x7fdb44073eb0 osd=2 4 2024-01-21T05:05:50.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.008+0000 7fdb567fc640 15 client.17463.objecter _session_command_op_remove 2 2 2024-01-21T05:05:50.009 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-21T05:05:50.010 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T05:05:50.010 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-21T05:05:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.008+0000 7fdb67fd0640 10 librados: watch_flush enter 2024-01-21T05:05:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 10 librados: watch_flush exit 2024-01-21T05:05:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 10 client.17463.objecter close_session for osd.2 2024-01-21T05:05:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 1 -- 172.21.15.69:0/3113459401 >> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] conn(0x7fdb44074580 msgr2=0x7fdb44076a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.69:6816/3954012353,v1:172.21.15.69:6817/3954012353] conn(0x7fdb44074580 0x7fdb44076a20 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 20 client.17463.objecter put_session s=0x7fdb44073eb0 osd=2 1 2024-01-21T05:05:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 20 client.17463.objecter shutdown clearing up homeless session... 2024-01-21T05:05:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 10 client.17463.objecter successfully canceled tick 2024-01-21T05:05:50.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 1 -- 172.21.15.69:0/3113459401 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdb4406c890 msgr2=0x7fdb4406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdb4406c890 0x7fdb4406ed50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdb50006fd0 tx=0x7fdb50008040 comp rx=0 tx=0).stop 2024-01-21T05:05:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 10 monclient: shutdown 2024-01-21T05:05:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 1 -- 172.21.15.69:0/3113459401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb600a8a40 msgr2=0x7fdb600ab6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 1 --2- 172.21.15.69:0/3113459401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb600a8a40 0x7fdb600ab6e0 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c000c00 tx=0x7fdb5c0046c0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb6656d640 1 -- 172.21.15.69:0/3113459401 reap_dead start 2024-01-21T05:05:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 1 -- 172.21.15.69:0/3113459401 shutdown_connections 2024-01-21T05:05:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 1 -- 172.21.15.69:0/3113459401 >> 172.21.15.69:0/3113459401 conn(0x7fdb60066f10 msgr2=0x7fdb6013b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 1 -- 172.21.15.69:0/3113459401 shutdown_connections 2024-01-21T05:05:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 1 -- 172.21.15.69:0/3113459401 wait complete. 2024-01-21T05:05:50.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.009+0000 7fdb67fd0640 1 librados: shutdown 2024-01-21T05:05:50.021 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-21T05:05:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.177+0000 7f02f80da640 10 monclient: get_monmap_and_config 2024-01-21T05:05:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.177+0000 7f02f80da640 10 monclient: build_initial_monmap 2024-01-21T05:05:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.177+0000 7f02f80da640 10 monclient: monmap: 2024-01-21T05:05:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:50.178637+0000 2024-01-21T05:05:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:50.178637+0000 2024-01-21T05:05:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:50.179 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.178+0000 7f02f80da640 1 Processor -- start 2024-01-21T05:05:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.178+0000 7f02f80da640 1 -- start start 2024-01-21T05:05:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.178+0000 7f02f80da640 10 monclient: init 2024-01-21T05:05:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-01-21T05:05:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02f00a8b20 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f00a9d10 0x7f02f013f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 10 monclient(hunting): picked mon.noname-b con 0x7f02f00a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f02f00b2310 0x7f02f013f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 10 monclient(hunting): picked mon.noname-c con 0x7f02f00b2310 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f01456f0 con 0x7f02f00a8b20 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f00a98c0 con 0x7f02f00b2310 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f013fb00 con 0x7f02f00a9d10 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f80da640 10 monclient(hunting): authenticate will time out at 3308.520320s 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.179+0000 7f02f6677640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f02f00b2310 0x7f02f013f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f5675640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f00a9d10 0x7f02f013f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f6677640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f02f00b2310 0x7f02f013f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:41298/0 (socket says 172.21.15.69:41298) 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f6677640 1 -- 172.21.15.69:0/4011790264 learned_addr learned my addr 172.21.15.69:0/4011790264 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f6677640 10 monclient(hunting): get_auth_request con 0x7f02f00b2310 auth_method 0 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f6677640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f6677640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f6677640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f5675640 10 monclient(hunting): get_auth_request con 0x7f02f00a9d10 auth_method 0 2024-01-21T05:05:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f5675640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f5675640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f5675640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f6677640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f6677640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.180+0000 7f02f6677640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.181+0000 7f02f5675640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.181+0000 7f02f5675640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.181+0000 7f02f5675640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.182+0000 7f02f6677640 10 monclient(hunting): handle_auth_done global_id 21571 payload 274 2024-01-21T05:05:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.182+0000 7f02f6677640 1 -- 172.21.15.69:0/4011790264 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f00a9d10 msgr2=0x7f02f013f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.182+0000 7f02f6677640 1 --2- 172.21.15.69:0/4011790264 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f00a9d10 0x7f02f013f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.182+0000 7f02f6677640 1 -- 172.21.15.69:0/4011790264 >> v1:172.21.15.69:6789/0 conn(0x7f02f00a8b20 legacy=0x7f02f0145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.182+0000 7f02f6677640 10 monclient: _finish_hunting 0 2024-01-21T05:05:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.182+0000 7f02f6677640 1 monclient: found mon.noname-c 2024-01-21T05:05:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.182+0000 7f02f6677640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.182+0000 7f02f6677640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T05:05:50.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.182+0000 7f02f6677640 1 -- 172.21.15.69:0/4011790264 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f0146290 con 0x7f02f00b2310 2024-01-21T05:05:50.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.184+0000 7f02f6677640 1 --2- 172.21.15.69:0/4011790264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f00b2310 0x7f02f013f5c0 secure :-1 s=READY pgs=5224 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009e50 tx=0x7f02e802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=162720bc6d11ce5 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 1 -- 172.21.15.69:0/4011790264 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e802ca70 con 0x7f02f00b2310 2024-01-21T05:05:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:05:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 10 monclient: dump: 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 10 monclient: _finish_auth 0 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 10 monclient: _check_auth_tickets 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 1 -- 172.21.15.69:0/4011790264 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e802cbd0 con 0x7f02f00b2310 2024-01-21T05:05:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f80da640 5 monclient: authenticate success, global_id 21571 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f80da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f80da640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f80da640 4 set_mon_vals no callback set 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 1 -- 172.21.15.69:0/4011790264 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e80343f0 con 0x7f02f00b2310 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.188+0000 7f02f4e74640 10 monclient: dump: 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.192 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:50.192 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:50.192 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:50.192 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.189+0000 7f02f80da640 10 monclient: shutdown 2024-01-21T05:05:50.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.189+0000 7f02f80da640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:50.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.189+0000 7f02f80da640 1 -- 172.21.15.69:0/4011790264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f00b2310 msgr2=0x7f02f013f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.189+0000 7f02f80da640 1 --2- 172.21.15.69:0/4011790264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f00b2310 0x7f02f013f5c0 secure :-1 s=READY pgs=5224 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009e50 tx=0x7f02e802bed0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.189+0000 7f02f80da640 1 -- 172.21.15.69:0/4011790264 shutdown_connections 2024-01-21T05:05:50.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.189+0000 7f02f80da640 1 --2- 172.21.15.69:0/4011790264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f00b2310 0x7f02f013f5c0 unknown :-1 s=CLOSED pgs=5224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.189+0000 7f02f80da640 1 --2- 172.21.15.69:0/4011790264 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f00a9d10 0x7f02f013f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.189+0000 7f02f80da640 1 -- 172.21.15.69:0/4011790264 >> 172.21.15.69:0/4011790264 conn(0x7f02f0066fb0 msgr2=0x7f02f00b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:50.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.189+0000 7f02f80da640 1 -- 172.21.15.69:0/4011790264 shutdown_connections 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.189+0000 7f02f80da640 1 -- 172.21.15.69:0/4011790264 wait complete. 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.190+0000 7f02f80da640 10 monclient: build_initial_monmap 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.190+0000 7f02f80da640 10 monclient: monmap: 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.190+0000 7f02f80da640 1 librados: starting msgr at 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.190+0000 7f02f80da640 1 librados: starting objecter 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.190+0000 7f02f80da640 1 Processor -- start 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.190+0000 7f02f80da640 1 -- start start 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.190+0000 7f02f80da640 1 librados: setting wanted keys 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.190+0000 7f02f80da640 1 librados: calling monclient init 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.190+0000 7f02f80da640 10 monclient: init 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f00a8b20 0x7f02f0137f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 10 monclient(hunting): picked mon.noname-c con 0x7f02f00a8b20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f00a9d10 0x7f02f0140280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 10 monclient(hunting): picked mon.noname-b con 0x7f02f00a9d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f00b2310 0x7f02f01407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 10 monclient(hunting): picked mon.noname-a con 0x7f02f00b2310 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:50.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f01391f0 con 0x7f02f00b2310 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f01394d0 con 0x7f02f00a9d10 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f013fd60 con 0x7f02f00a8b20 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f80da640 10 monclient(hunting): authenticate will time out at 3308.532383s 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f6677640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f00b2310 0x7f02f01407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f5e76640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f00a8b20 0x7f02f0137f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f5675640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f00a9d10 0x7f02f0140280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.191+0000 7f02f5675640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f00a9d10 0x7f02f0140280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:41304/0 (socket says 172.21.15.69:41304) 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5675640 1 -- 172.21.15.69:0/4265323693 learned_addr learned my addr 172.21.15.69:0/4265323693 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5675640 10 monclient(hunting): get_auth_request con 0x7f02f00a9d10 auth_method 0 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5675640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5675640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:50.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5675640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5e76640 10 monclient(hunting): get_auth_request con 0x7f02f00a8b20 auth_method 0 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5e76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5e76640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5e76640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f6677640 10 monclient(hunting): get_auth_request con 0x7f02f00b2310 auth_method 0 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f6677640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f6677640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f6677640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5675640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5675640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5675640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5e76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5e76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f5e76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f6677640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:50.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f6677640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:50.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.192+0000 7f02f6677640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:50.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.195+0000 7f02f5675640 10 monclient(hunting): handle_auth_done global_id 21574 payload 824 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.195+0000 7f02f5675640 1 -- 172.21.15.69:0/4265323693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f00a8b20 msgr2=0x7f02f0137f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.195+0000 7f02f5675640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f00a8b20 0x7f02f0137f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.195+0000 7f02f5675640 1 -- 172.21.15.69:0/4265323693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f00b2310 msgr2=0x7f02f01407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.195+0000 7f02f5675640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f00b2310 0x7f02f01407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.195+0000 7f02f5675640 10 monclient: _finish_hunting 0 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.195+0000 7f02f5675640 1 monclient: found mon.noname-b 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.195+0000 7f02f5675640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.195+0000 7f02f5675640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.195+0000 7f02f5675640 1 -- 172.21.15.69:0/4265323693 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e8009b30 con 0x7f02f00a9d10 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.197+0000 7f02f5675640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f00a9d10 0x7f02f0140280 secure :-1 s=READY pgs=5225 cs=0 l=1 rev1=1 crypto rx=0x7f02e000e980 tx=0x7f02e000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.197+0000 7f02e67fc640 10 client.?.objecter ms_handle_connect 0x7f02f00a9d10 2024-01-21T05:05:50.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.197+0000 7f02e67fc640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:50.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 1 -- 172.21.15.69:0/4265323693 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0009670 con 0x7f02f00a9d10 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 10 monclient: dump: 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:50.203 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 10 monclient: _finish_auth 0 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 10 monclient: _check_auth_tickets 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 1 -- 172.21.15.69:0/4265323693 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e000c9a0 con 0x7f02f00a9d10 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 1 -- 172.21.15.69:0/4265323693 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e000cd30 con 0x7f02f00a9d10 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e67fc640 10 monclient: dump: 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.204 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02f80da640 5 monclient: authenticate success, global_id 21574 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02f80da640 10 monclient: _renew_subs 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02f80da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02f80da640 1 -- 172.21.15.69:0/4265323693 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f0138460 con 0x7f02f00a9d10 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.200+0000 7f02e7fff640 4 set_mon_vals no callback set 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.201+0000 7f02f80da640 10 client.21574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.201+0000 7f02f80da640 10 monclient: _renew_subs 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.201+0000 7f02f80da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:50.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.201+0000 7f02f80da640 1 -- 172.21.15.69:0/4265323693 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f0142780 con 0x7f02f00a9d10 2024-01-21T05:05:50.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.201+0000 7f02f80da640 1 librados: init done 2024-01-21T05:05:50.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.201+0000 7f02f80da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:50.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.201+0000 7f02f80da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:50.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.201+0000 7f02f80da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:50.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.201+0000 7f02f80da640 1 -- 172.21.15.69:0/4265323693 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f013ff20 con 0x7f02f00a9d10 2024-01-21T05:05:50.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.224+0000 7f02e67fc640 1 -- 172.21.15.69:0/4265323693 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f02e0013470 con 0x7f02f00a9d10 2024-01-21T05:05:50.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.224+0000 7f02e67fc640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f02d406c8b0 0x7f02d406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.224+0000 7f02f5e76640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f02d406c8b0 0x7f02d406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.224+0000 7f02f5e76640 10 monclient: get_auth_request con 0x7f02d406c8b0 auth_method 0 2024-01-21T05:05:50.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.225+0000 7f02f5e76640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f02d406c8b0 0x7f02d406ed70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f02ec004f60 tx=0x7f02ec007e30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:50.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.225+0000 7f02e67fc640 10 client.21574.objecter ms_handle_connect 0x7f02d406c8b0 2024-01-21T05:05:50.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.233+0000 7f02e67fc640 1 -- 172.21.15.69:0/4265323693 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f02e0058bb0 con 0x7f02f00a9d10 2024-01-21T05:05:50.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.233+0000 7f02e67fc640 10 client.21574.objecter ms_dispatch 0x7f02f0000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:50.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.233+0000 7f02e67fc640 3 client.21574.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:50.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.233+0000 7f02e67fc640 3 client.21574.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:50.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.233+0000 7f02e67fc640 20 client.21574.objecter dump_active .. 0 homeless 2024-01-21T05:05:50.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.467+0000 7f02e67fc640 1 -- 172.21.15.69:0/4265323693 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f02e0054ef0 con 0x7f02f00a9d10 2024-01-21T05:05:50.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.467+0000 7f02e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:50.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.467+0000 7f02e67fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:50.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.674+0000 7f02f80da640 1 -- 172.21.15.69:0/4265323693 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f02f004e970 con 0x7f02d406c8b0 2024-01-21T05:05:50.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.675+0000 7f02e67fc640 1 -- 172.21.15.69:0/4265323693 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f02f004e970 con 0x7f02d406c8b0 2024-01-21T05:05:50.676 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 2 to deep-scrub 2024-01-21T05:05:50.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 10 librados: watch_flush enter 2024-01-21T05:05:50.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 10 librados: watch_flush exit 2024-01-21T05:05:50.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 20 client.21574.objecter shutdown clearing up homeless session... 2024-01-21T05:05:50.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 10 client.21574.objecter successfully canceled tick 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 -- 172.21.15.69:0/4265323693 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f02d406c8b0 msgr2=0x7f02d406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f02d406c8b0 0x7f02d406ed70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f02ec004f60 tx=0x7f02ec007e30 comp rx=0 tx=0).stop 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 10 monclient: shutdown 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 -- 172.21.15.69:0/4265323693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f00a9d10 msgr2=0x7f02f0140280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f00a9d10 0x7f02f0140280 secure :-1 s=READY pgs=5225 cs=0 l=1 rev1=1 crypto rx=0x7f02e000e980 tx=0x7f02e000ee40 comp rx=0 tx=0).stop 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 -- 172.21.15.69:0/4265323693 shutdown_connections 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f02d406c8b0 0x7f02d406ed70 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02f00b2310 0x7f02f01407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02f00a9d10 0x7f02f0140280 unknown :-1 s=CLOSED pgs=5225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 --2- 172.21.15.69:0/4265323693 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f02f00a8b20 0x7f02f0137f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 -- 172.21.15.69:0/4265323693 >> 172.21.15.69:0/4265323693 conn(0x7f02f0066fb0 msgr2=0x7f02f00b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 -- 172.21.15.69:0/4265323693 shutdown_connections 2024-01-21T05:05:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 -- 172.21.15.69:0/4265323693 wait complete. 2024-01-21T05:05:50.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.677+0000 7f02f80da640 1 librados: shutdown 2024-01-21T05:05:50.689 INFO:tasks.ceph:Scrubbing osd.3 2024-01-21T05:05:50.689 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-21T05:05:50.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.792+0000 7fdb2979e640 10 monclient: get_monmap_and_config 2024-01-21T05:05:50.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.792+0000 7fdb2979e640 10 monclient: build_initial_monmap 2024-01-21T05:05:50.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.792+0000 7fdb2979e640 10 monclient: monmap: 2024-01-21T05:05:50.793 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:50.793 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:50.793 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:50.793808+0000 2024-01-21T05:05:50.793 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:50.793808+0000 2024-01-21T05:05:50.793 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:50.794 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.794 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:50.794 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:50.794 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:50.794 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:50.794 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:50.794 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.793+0000 7fdb2979e640 1 Processor -- start 2024-01-21T05:05:50.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.793+0000 7fdb2979e640 1 -- start start 2024-01-21T05:05:50.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.793+0000 7fdb2979e640 10 monclient: init 2024-01-21T05:05:50.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb240a8a40 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdb240b2210 0x7fdb2413f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 10 monclient(hunting): picked mon.noname-c con 0x7fdb240b2210 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8e20 0x7fdb2413f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 10 monclient(hunting): picked mon.noname-b con 0x7fdb240a8e20 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2413fc40 con 0x7fdb240a8a40 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb240aa7b0 con 0x7fdb240b2210 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24145700 con 0x7fdb240a8e20 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:50.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb2979e640 10 monclient(hunting): authenticate will time out at 3309.135302s 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb237fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8e20 0x7fdb2413f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.794+0000 7fdb227fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fdb240b2210 0x7fdb2413f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb237fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8e20 0x7fdb2413f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:48616/0 (socket says 172.21.15.69:48616) 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb237fe640 1 -- 172.21.15.69:0/2392509121 learned_addr learned my addr 172.21.15.69:0/2392509121 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb237fe640 10 monclient(hunting): get_auth_request con 0x7fdb240a8e20 auth_method 0 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb237fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb237fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb227fc640 10 monclient(hunting): get_auth_request con 0x7fdb240b2210 auth_method 0 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb227fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:50.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb227fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:50.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:50.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:50.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:50.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:50.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:50.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.795+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:50.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.797+0000 7fdb237fe640 10 monclient(hunting): handle_auth_done global_id 18848 payload 274 2024-01-21T05:05:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.797+0000 7fdb237fe640 1 -- 172.21.15.69:0/2392509121 >> v2:172.21.15.144:3300/0 conn(0x7fdb240b2210 msgr2=0x7fdb2413f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.797+0000 7fdb237fe640 1 --2- 172.21.15.69:0/2392509121 >> v2:172.21.15.144:3300/0 conn(0x7fdb240b2210 0x7fdb2413f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.797+0000 7fdb237fe640 1 -- 172.21.15.69:0/2392509121 >> v1:172.21.15.144:6789/0 conn(0x7fdb240a8a40 legacy=0x7fdb24145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.797+0000 7fdb237fe640 10 monclient: _finish_hunting 0 2024-01-21T05:05:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.797+0000 7fdb237fe640 1 monclient: found mon.noname-b 2024-01-21T05:05:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.797+0000 7fdb237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.797+0000 7fdb237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T05:05:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.797+0000 7fdb237fe640 1 -- 172.21.15.69:0/2392509121 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb240aa9b0 con 0x7fdb240a8e20 2024-01-21T05:05:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.799+0000 7fdb237fe640 1 --2- 172.21.15.69:0/2392509121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8e20 0x7fdb2413f700 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009e50 tx=0x7fdb1c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcfa817b50ae6f3c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:50.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 1 -- 172.21.15.69:0/2392509121 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c02cb00 con 0x7fdb240a8e20 2024-01-21T05:05:50.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:50.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:50.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 10 monclient: dump: 2024-01-21T05:05:50.803 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:50.803 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:50.803 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 10 monclient: _finish_auth 0 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 10 monclient: _check_auth_tickets 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 1 -- 172.21.15.69:0/2392509121 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c02cc60 con 0x7fdb240a8e20 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 1 -- 172.21.15.69:0/2392509121 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c0343f0 con 0x7fdb240a8e20 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb2979e640 5 monclient: authenticate success, global_id 18848 2024-01-21T05:05:50.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb2979e640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb2979e640 4 set_mon_vals no callback set 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.802+0000 7fdb21ffb640 10 monclient: dump: 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:50.805 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.803+0000 7fdb2979e640 10 monclient: shutdown 2024-01-21T05:05:50.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.803+0000 7fdb2979e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:50.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.803+0000 7fdb2979e640 1 -- 172.21.15.69:0/2392509121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8e20 msgr2=0x7fdb2413f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.803+0000 7fdb2979e640 1 --2- 172.21.15.69:0/2392509121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8e20 0x7fdb2413f700 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009e50 tx=0x7fdb1c02bf60 comp rx=0 tx=0).stop 2024-01-21T05:05:50.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.803+0000 7fdb2979e640 1 -- 172.21.15.69:0/2392509121 shutdown_connections 2024-01-21T05:05:50.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.803+0000 7fdb2979e640 1 --2- 172.21.15.69:0/2392509121 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8e20 0x7fdb2413f700 unknown :-1 s=CLOSED pgs=3325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.803+0000 7fdb2979e640 1 --2- 172.21.15.69:0/2392509121 >> v2:172.21.15.144:3300/0 conn(0x7fdb240b2210 0x7fdb2413f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.803+0000 7fdb2979e640 1 -- 172.21.15.69:0/2392509121 >> 172.21.15.69:0/2392509121 conn(0x7fdb24066f10 msgr2=0x7fdb240b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.803+0000 7fdb2979e640 1 -- 172.21.15.69:0/2392509121 shutdown_connections 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.803+0000 7fdb2979e640 1 -- 172.21.15.69:0/2392509121 wait complete. 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.804+0000 7fdb2979e640 10 monclient: build_initial_monmap 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.804+0000 7fdb2979e640 10 monclient: monmap: 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.804+0000 7fdb2979e640 1 librados: starting msgr at 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.804+0000 7fdb2979e640 1 librados: starting objecter 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.804+0000 7fdb2979e640 1 Processor -- start 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.804+0000 7fdb2979e640 1 -- start start 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.804+0000 7fdb2979e640 1 librados: setting wanted keys 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.804+0000 7fdb2979e640 1 librados: calling monclient init 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.804+0000 7fdb2979e640 10 monclient: init 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8a40 0x7fdb2414a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 10 monclient(hunting): picked mon.noname-c con 0x7fdb240a8a40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb240a8e20 0x7fdb241494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 10 monclient(hunting): picked mon.noname-a con 0x7fdb240a8e20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb240b2210 0x7fdb241da600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 10 monclient(hunting): picked mon.noname-b con 0x7fdb240b2210 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24149a20 con 0x7fdb240a8e20 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb24149d60 con 0x7fdb240b2210 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb241dab40 con 0x7fdb240a8a40 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb2979e640 10 monclient(hunting): authenticate will time out at 3309.146661s 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.805+0000 7fdb22ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8a40 0x7fdb2414a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb227fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb240a8e20 0x7fdb241494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb22ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8a40 0x7fdb2414a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:48628/0 (socket says 172.21.15.69:48628) 2024-01-21T05:05:50.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb22ffd640 1 -- 172.21.15.69:0/4162947480 learned_addr learned my addr 172.21.15.69:0/4162947480 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb22ffd640 10 monclient(hunting): get_auth_request con 0x7fdb240a8a40 auth_method 0 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb22ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb227fc640 10 monclient(hunting): get_auth_request con 0x7fdb240a8e20 auth_method 0 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb227fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb227fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.806+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.807+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.807+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:50.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.807+0000 7fdb227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:50.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.809+0000 7fdb22ffd640 10 monclient(hunting): handle_auth_done global_id 18851 payload 824 2024-01-21T05:05:50.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.809+0000 7fdb22ffd640 1 -- 172.21.15.69:0/4162947480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb240b2210 msgr2=0x7fdb241da600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.809+0000 7fdb22ffd640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdb240b2210 0x7fdb241da600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.809+0000 7fdb22ffd640 1 -- 172.21.15.69:0/4162947480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb240a8e20 msgr2=0x7fdb241494e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.809+0000 7fdb22ffd640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb240a8e20 0x7fdb241494e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.809+0000 7fdb22ffd640 10 monclient: _finish_hunting 0 2024-01-21T05:05:50.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.809+0000 7fdb22ffd640 1 monclient: found mon.noname-c 2024-01-21T05:05:50.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.809+0000 7fdb22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:50.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.809+0000 7fdb22ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T05:05:50.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.809+0000 7fdb22ffd640 1 -- 172.21.15.69:0/4162947480 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c009b30 con 0x7fdb240a8a40 2024-01-21T05:05:50.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.812+0000 7fdb22ffd640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8a40 0x7fdb2414a280 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400c430 tx=0x7fdb1400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:50.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.812+0000 7fdb0b7fe640 10 client.?.objecter ms_handle_connect 0x7fdb240a8a40 2024-01-21T05:05:50.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.812+0000 7fdb0b7fe640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.813+0000 7fdb0b7fe640 1 -- 172.21.15.69:0/4162947480 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb140052d0 con 0x7fdb240a8a40 2024-01-21T05:05:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.813+0000 7fdb0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.813+0000 7fdb0b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:05:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.813+0000 7fdb0b7fe640 10 monclient: dump: 2024-01-21T05:05:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.813+0000 7fdb0b7fe640 10 monclient: _finish_auth 0 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.813+0000 7fdb0b7fe640 10 monclient: _check_auth_tickets 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.813+0000 7fdb0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.813+0000 7fdb2979e640 5 monclient: authenticate success, global_id 18851 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.813+0000 7fdb2979e640 10 monclient: _renew_subs 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.813+0000 7fdb2979e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.813+0000 7fdb2979e640 1 -- 172.21.15.69:0/4162947480 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb241dadc0 con 0x7fdb240a8a40 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.814+0000 7fdb2979e640 10 client.18851.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.814+0000 7fdb2979e640 10 monclient: _renew_subs 2024-01-21T05:05:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.814+0000 7fdb2979e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T05:05:50.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.814+0000 7fdb2979e640 1 -- 172.21.15.69:0/4162947480 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb24147a50 con 0x7fdb240a8a40 2024-01-21T05:05:50.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.814+0000 7fdb2979e640 1 librados: init done 2024-01-21T05:05:50.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.814+0000 7fdb2979e640 10 client.18851.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:50.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.814+0000 7fdb2979e640 15 client.18851.objecter _session_command_op_assign -1 1 2024-01-21T05:05:50.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.814+0000 7fdb2979e640 20 client.18851.objecter _get_session osd=-1 returning homeless 2024-01-21T05:05:50.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.814+0000 7fdb2979e640 10 client.18851.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:50.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.814+0000 7fdb2979e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T05:05:50.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.814+0000 7fdb2979e640 1 -- 172.21.15.69:0/4162947480 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb240aafb0 con 0x7fdb240a8a40 2024-01-21T05:05:50.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.815+0000 7fdb0b7fe640 1 -- 172.21.15.69:0/4162947480 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb14005430 con 0x7fdb240a8a40 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.815+0000 7fdb0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.815+0000 7fdb0b7fe640 1 -- 172.21.15.69:0/4162947480 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14007220 con 0x7fdb240a8a40 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.815+0000 7fdb0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.815+0000 7fdb0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.815+0000 7fdb0b7fe640 10 monclient: dump: 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:50.818 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.815+0000 7fdb217fa640 4 set_mon_vals no callback set 2024-01-21T05:05:50.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.836+0000 7fdb0b7fe640 1 -- 172.21.15.69:0/4162947480 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fdb14007380 con 0x7fdb240a8a40 2024-01-21T05:05:50.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.836+0000 7fdb0b7fe640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdb1006c860 0x7fdb1006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.837+0000 7fdb227fc640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdb1006c860 0x7fdb1006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.837+0000 7fdb227fc640 10 monclient: get_auth_request con 0x7fdb1006c860 auth_method 0 2024-01-21T05:05:50.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.837+0000 7fdb227fc640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdb1006c860 0x7fdb1006ed20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdb2414ad80 tx=0x7fdb18009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:50.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.837+0000 7fdb0b7fe640 10 client.18851.objecter ms_handle_connect 0x7fdb1006c860 2024-01-21T05:05:50.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 1 -- 172.21.15.69:0/4162947480 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7fdb14004f10 con 0x7fdb240a8a40 2024-01-21T05:05:50.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 10 client.18851.objecter ms_dispatch 0x7fdb24000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:50.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 3 client.18851.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:50.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 3 client.18851.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:50.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 10 client.18851.objecter checking command 1 2024-01-21T05:05:50.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] conn(0x7fdb10074550 0x7fdb100769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:50.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 20 client.18851.objecter _get_session s=0x7fdb10073e80 osd=3 3 2024-01-21T05:05:50.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 20 client.18851.objecter put_session s=0x7fdb10073e80 osd=3 3 2024-01-21T05:05:50.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 15 client.18851.objecter _session_command_op_remove -1 1 2024-01-21T05:05:50.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 20 client.18851.objecter _get_session s=0x7fdb10073e80 osd=3 3 2024-01-21T05:05:50.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 20 client.18851.objecter get_session s=0x7fdb10073e80 osd=3 3 2024-01-21T05:05:50.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 15 client.18851.objecter _session_command_op_assign 3 1 2024-01-21T05:05:50.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 20 client.18851.objecter put_session s=0x7fdb10073e80 osd=3 4 2024-01-21T05:05:50.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 10 client.18851.objecter _send_command 1 2024-01-21T05:05:50.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 1 -- 172.21.15.69:0/4162947480 --> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb100770c0 con 0x7fdb10074550 2024-01-21T05:05:50.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 20 client.18851.objecter dump_active .. 0 homeless 2024-01-21T05:05:50.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 1 -- 172.21.15.69:0/4162947480 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_get_version_reply(handle=1 version=1109) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb14053ee0 con 0x7fdb240a8a40 2024-01-21T05:05:50.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb0b7fe640 10 monclient: handle_get_version_reply finishing 1 version 1109 2024-01-21T05:05:50.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb237fe640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] conn(0x7fdb10074550 0x7fdb100769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:50.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.839+0000 7fdb237fe640 10 monclient: get_auth_request con 0x7fdb10074550 auth_method 0 2024-01-21T05:05:50.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.842+0000 7fdb237fe640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] conn(0x7fdb10074550 0x7fdb100769f0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:50.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.842+0000 7fdb0b7fe640 10 client.18851.objecter ms_handle_connect 0x7fdb10074550 2024-01-21T05:05:50.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.866+0000 7fdb0b7fe640 1 -- 172.21.15.69:0/4162947480 <== osd.3 v2:172.21.15.69:6800/1086763384 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdb100770c0 con 0x7fdb10074550 2024-01-21T05:05:50.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.866+0000 7fdb0b7fe640 10 client.18851.objecter ms_dispatch 0x7fdb24000b90 command_reply(tid 1: 0 ) v1 2024-01-21T05:05:50.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.866+0000 7fdb0b7fe640 10 client.18851.objecter _finish_command 1 = system:0 2024-01-21T05:05:50.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.866+0000 7fdb0b7fe640 20 client.18851.objecter put_session s=0x7fdb10073e80 osd=3 4 2024-01-21T05:05:50.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.866+0000 7fdb0b7fe640 15 client.18851.objecter _session_command_op_remove 3 1 2024-01-21T05:05:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.879+0000 7fdb2979e640 10 client.18851.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T05:05:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.879+0000 7fdb2979e640 15 client.18851.objecter _session_command_op_assign -1 2 2024-01-21T05:05:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.879+0000 7fdb2979e640 20 client.18851.objecter _get_session s=0x7fdb10073e80 osd=3 3 2024-01-21T05:05:50.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.879+0000 7fdb2979e640 20 client.18851.objecter put_session s=0x7fdb10073e80 osd=3 3 2024-01-21T05:05:50.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.879+0000 7fdb2979e640 20 client.18851.objecter _get_session s=0x7fdb10073e80 osd=3 3 2024-01-21T05:05:50.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.879+0000 7fdb2979e640 15 client.18851.objecter _session_command_op_remove -1 2 2024-01-21T05:05:50.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.879+0000 7fdb2979e640 20 client.18851.objecter get_session s=0x7fdb10073e80 osd=3 3 2024-01-21T05:05:50.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.879+0000 7fdb2979e640 15 client.18851.objecter _session_command_op_assign 3 2 2024-01-21T05:05:50.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.879+0000 7fdb2979e640 20 client.18851.objecter put_session s=0x7fdb10073e80 osd=3 4 2024-01-21T05:05:50.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.879+0000 7fdb2979e640 10 client.18851.objecter _send_command 2 2024-01-21T05:05:50.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.879+0000 7fdb2979e640 1 -- 172.21.15.69:0/4162947480 --> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdb240aa1c0 con 0x7fdb10074550 2024-01-21T05:05:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb0b7fe640 1 -- 172.21.15.69:0/4162947480 <== osd.3 v2:172.21.15.69:6800/1086763384 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fdb240aa1c0 con 0x7fdb10074550 2024-01-21T05:05:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb0b7fe640 10 client.18851.objecter ms_dispatch 0x7fdb24000b90 command_reply(tid 2: 0 ) v1 2024-01-21T05:05:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb0b7fe640 10 client.18851.objecter _finish_command 2 = system:0 2024-01-21T05:05:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb0b7fe640 20 client.18851.objecter put_session s=0x7fdb10073e80 osd=3 4 2024-01-21T05:05:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb0b7fe640 15 client.18851.objecter _session_command_op_remove 3 2 2024-01-21T05:05:50.894 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-21T05:05:50.894 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T05:05:50.894 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-21T05:05:50.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb2979e640 10 librados: watch_flush enter 2024-01-21T05:05:50.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb2979e640 10 librados: watch_flush exit 2024-01-21T05:05:50.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb2979e640 10 client.18851.objecter close_session for osd.3 2024-01-21T05:05:50.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb2979e640 1 -- 172.21.15.69:0/4162947480 >> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] conn(0x7fdb10074550 msgr2=0x7fdb100769f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb2979e640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.69:6800/1086763384,v1:172.21.15.69:6801/1086763384] conn(0x7fdb10074550 0x7fdb100769f0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb2979e640 20 client.18851.objecter put_session s=0x7fdb10073e80 osd=3 1 2024-01-21T05:05:50.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb2979e640 20 client.18851.objecter shutdown clearing up homeless session... 2024-01-21T05:05:50.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.892+0000 7fdb2979e640 10 client.18851.objecter successfully canceled tick 2024-01-21T05:05:50.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb2979e640 1 -- 172.21.15.69:0/4162947480 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdb1006c860 msgr2=0x7fdb1006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb2979e640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fdb1006c860 0x7fdb1006ed20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdb2414ad80 tx=0x7fdb18009190 comp rx=0 tx=0).stop 2024-01-21T05:05:50.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb2979e640 10 monclient: shutdown 2024-01-21T05:05:50.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb2979e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:50.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb2979e640 1 -- 172.21.15.69:0/4162947480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8a40 msgr2=0x7fdb2414a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:50.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb2979e640 1 --2- 172.21.15.69:0/4162947480 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fdb240a8a40 0x7fdb2414a280 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7fdb1400c430 tx=0x7fdb1400c8f0 comp rx=0 tx=0).stop 2024-01-21T05:05:50.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb237fe640 1 -- 172.21.15.69:0/4162947480 reap_dead start 2024-01-21T05:05:50.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb2979e640 1 -- 172.21.15.69:0/4162947480 shutdown_connections 2024-01-21T05:05:50.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb2979e640 1 -- 172.21.15.69:0/4162947480 >> 172.21.15.69:0/4162947480 conn(0x7fdb24066f10 msgr2=0x7fdb2413e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:50.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb2979e640 1 -- 172.21.15.69:0/4162947480 shutdown_connections 2024-01-21T05:05:50.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb2979e640 1 -- 172.21.15.69:0/4162947480 wait complete. 2024-01-21T05:05:50.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:50.893+0000 7fdb2979e640 1 librados: shutdown 2024-01-21T05:05:50.904 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-21T05:05:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.014+0000 7ff47f44f640 10 monclient: get_monmap_and_config 2024-01-21T05:05:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.014+0000 7ff47f44f640 10 monclient: build_initial_monmap 2024-01-21T05:05:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.014+0000 7ff47f44f640 10 monclient: monmap: 2024-01-21T05:05:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:51.015840+0000 2024-01-21T05:05:51.016 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:51.015840+0000 2024-01-21T05:05:51.016 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:51.016 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.016 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:51.016 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:51.016 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:51.016 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:51.016 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:51.016 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.015+0000 7ff47f44f640 1 Processor -- start 2024-01-21T05:05:51.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.015+0000 7ff47f44f640 1 -- start start 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.015+0000 7ff47f44f640 10 monclient: init 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4780a8b20 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff4780a9d10 0x7ff47813f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 10 monclient(hunting): picked mon.noname-a con 0x7ff4780a9d10 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff4780b2310 0x7ff47813f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 10 monclient(hunting): picked mon.noname-c con 0x7ff4780b2310 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4781456f0 con 0x7ff4780a8b20 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4780a98c0 con 0x7ff4780a9d10 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff47813fb00 con 0x7ff4780b2310 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:51.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47f44f640 10 monclient(hunting): authenticate will time out at 3309.357498s 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.016+0000 7ff47c9ea640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff4780a9d10 0x7ff47813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47d9ec640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7ff4780b2310 0x7ff47813f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47c9ea640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff4780a9d10 0x7ff47813f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41682/0 (socket says 172.21.15.69:41682) 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47c9ea640 1 -- 172.21.15.69:0/2193919821 learned_addr learned my addr 172.21.15.69:0/2193919821 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47c9ea640 10 monclient(hunting): get_auth_request con 0x7ff4780a9d10 auth_method 0 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47c9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47c9ea640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47c9ea640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47d9ec640 10 monclient(hunting): get_auth_request con 0x7ff4780b2310 auth_method 0 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47d9ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47d9ec640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47d9ec640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47c9ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47c9ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.017+0000 7ff47c9ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.018+0000 7ff47d9ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.018+0000 7ff47d9ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.018+0000 7ff47d9ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.019+0000 7ff47c9ea640 10 monclient(hunting): handle_auth_done global_id 17472 payload 274 2024-01-21T05:05:51.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.019+0000 7ff47c9ea640 1 -- 172.21.15.69:0/2193919821 >> v2:172.21.15.144:3300/0 conn(0x7ff4780b2310 msgr2=0x7ff47813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:51.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.019+0000 7ff47c9ea640 1 --2- 172.21.15.69:0/2193919821 >> v2:172.21.15.144:3300/0 conn(0x7ff4780b2310 0x7ff47813f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.019+0000 7ff47c9ea640 1 -- 172.21.15.69:0/2193919821 >> v1:172.21.15.144:6789/0 conn(0x7ff4780a8b20 legacy=0x7ff478145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.019+0000 7ff47c9ea640 10 monclient: _finish_hunting 0 2024-01-21T05:05:51.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.019+0000 7ff47c9ea640 1 monclient: found mon.noname-a 2024-01-21T05:05:51.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.019+0000 7ff47c9ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:51.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.019+0000 7ff47c9ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:51.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.019+0000 7ff47c9ea640 1 -- 172.21.15.69:0/2193919821 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff478146290 con 0x7ff4780a9d10 2024-01-21T05:05:51.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.021+0000 7ff47c9ea640 1 --2- 172.21.15.69:0/2193919821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4780a9d10 0x7ff47813f080 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7ff4780aa150 tx=0x7ff46802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca8d6b94eb1bc6b server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 1 -- 172.21.15.69:0/2193919821 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468031870 con 0x7ff4780a9d10 2024-01-21T05:05:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 10 monclient: dump: 2024-01-21T05:05:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 10 monclient: _finish_auth 0 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 10 monclient: _check_auth_tickets 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 1 -- 172.21.15.69:0/2193919821 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4680319d0 con 0x7ff4780a9d10 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff47f44f640 5 monclient: authenticate success, global_id 17472 2024-01-21T05:05:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 1 -- 172.21.15.69:0/2193919821 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468031d60 con 0x7ff4780a9d10 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff46ffff640 10 monclient: dump: 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.028 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.029 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff47f44f640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:51.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.025+0000 7ff47f44f640 4 set_mon_vals no callback set 2024-01-21T05:05:51.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.026+0000 7ff47f44f640 10 monclient: shutdown 2024-01-21T05:05:51.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.026+0000 7ff47f44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:51.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.026+0000 7ff47f44f640 1 -- 172.21.15.69:0/2193919821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4780a9d10 msgr2=0x7ff47813f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.026+0000 7ff47f44f640 1 --2- 172.21.15.69:0/2193919821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4780a9d10 0x7ff47813f080 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7ff4780aa150 tx=0x7ff46802b910 comp rx=0 tx=0).stop 2024-01-21T05:05:51.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.026+0000 7ff47f44f640 1 -- 172.21.15.69:0/2193919821 shutdown_connections 2024-01-21T05:05:51.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.026+0000 7ff47f44f640 1 --2- 172.21.15.69:0/2193919821 >> v2:172.21.15.144:3300/0 conn(0x7ff4780b2310 0x7ff47813f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.026+0000 7ff47f44f640 1 --2- 172.21.15.69:0/2193919821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4780a9d10 0x7ff47813f080 unknown :-1 s=CLOSED pgs=3498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.026+0000 7ff47f44f640 1 -- 172.21.15.69:0/2193919821 >> 172.21.15.69:0/2193919821 conn(0x7ff478066fb0 msgr2=0x7ff4780b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:51.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.026+0000 7ff47f44f640 1 -- 172.21.15.69:0/2193919821 shutdown_connections 2024-01-21T05:05:51.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.026+0000 7ff47f44f640 1 -- 172.21.15.69:0/2193919821 wait complete. 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.027+0000 7ff47f44f640 10 monclient: build_initial_monmap 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.027+0000 7ff47f44f640 10 monclient: monmap: 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.027+0000 7ff47f44f640 1 librados: starting msgr at 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.027+0000 7ff47f44f640 1 librados: starting objecter 2024-01-21T05:05:51.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 1 Processor -- start 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 1 -- start start 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 1 librados: setting wanted keys 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 1 librados: calling monclient init 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 10 monclient: init 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4780a8b20 0x7ff478137f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 10 monclient(hunting): picked mon.noname-a con 0x7ff4780a8b20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4780a9d10 0x7ff4781402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 10 monclient(hunting): picked mon.noname-c con 0x7ff4780a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4780b2310 0x7ff4781407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 10 monclient(hunting): picked mon.noname-b con 0x7ff4780b2310 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff478138930 con 0x7ff4780a8b20 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff478139970 con 0x7ff4780b2310 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff478139c50 con 0x7ff4780a9d10 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.028+0000 7ff47f44f640 10 monclient(hunting): authenticate will time out at 3309.369916s 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47d1eb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4780a8b20 0x7ff478137f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47c9ea640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4780a9d10 0x7ff4781402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47d9ec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4780b2310 0x7ff4781407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47c9ea640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4780a9d10 0x7ff4781402a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:48640/0 (socket says 172.21.15.69:48640) 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47c9ea640 1 -- 172.21.15.69:0/3518222716 learned_addr learned my addr 172.21.15.69:0/3518222716 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47c9ea640 10 monclient(hunting): get_auth_request con 0x7ff4780a9d10 auth_method 0 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47c9ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47c9ea640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47c9ea640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47d1eb640 10 monclient(hunting): get_auth_request con 0x7ff4780a8b20 auth_method 0 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47d1eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47d1eb640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47d1eb640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47d9ec640 10 monclient(hunting): get_auth_request con 0x7ff4780b2310 auth_method 0 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47d9ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47d9ec640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.029+0000 7ff47d9ec640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.030+0000 7ff47c9ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.030+0000 7ff47c9ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.030+0000 7ff47c9ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.030+0000 7ff47d1eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.030+0000 7ff47d1eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.030+0000 7ff47d1eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.030+0000 7ff47d9ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.030+0000 7ff47d9ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.030+0000 7ff47d9ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.032+0000 7ff47d9ec640 10 monclient(hunting): handle_auth_done global_id 21586 payload 824 2024-01-21T05:05:51.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.032+0000 7ff47d9ec640 1 -- 172.21.15.69:0/3518222716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4780a9d10 msgr2=0x7ff4781402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.032+0000 7ff47d9ec640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4780a9d10 0x7ff4781402a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.032+0000 7ff47d9ec640 1 -- 172.21.15.69:0/3518222716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4780a8b20 msgr2=0x7ff478137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.032+0000 7ff47d9ec640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4780a8b20 0x7ff478137f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.032+0000 7ff47d9ec640 10 monclient: _finish_hunting 0 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.032+0000 7ff47d9ec640 1 monclient: found mon.noname-b 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.032+0000 7ff47d9ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.032+0000 7ff47d9ec640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.032+0000 7ff47d9ec640 1 -- 172.21.15.69:0/3518222716 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4680094d0 con 0x7ff4780b2310 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.034+0000 7ff47d9ec640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4780b2310 0x7ff4781407e0 secure :-1 s=READY pgs=5228 cs=0 l=1 rev1=1 crypto rx=0x7ff470009d90 tx=0x7ff47000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.034+0000 7ff46dffb640 10 client.?.objecter ms_handle_connect 0x7ff4780b2310 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.034+0000 7ff46dffb640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.036+0000 7ff46dffb640 1 -- 172.21.15.69:0/3518222716 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff470014480 con 0x7ff4780b2310 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.036+0000 7ff46dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.036+0000 7ff46dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:51.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.036+0000 7ff46dffb640 10 monclient: dump: 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.036+0000 7ff46dffb640 10 monclient: _finish_auth 0 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.036+0000 7ff46dffb640 10 monclient: _check_auth_tickets 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.036+0000 7ff46dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.036+0000 7ff47f44f640 5 monclient: authenticate success, global_id 21586 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.036+0000 7ff47f44f640 10 monclient: _renew_subs 2024-01-21T05:05:51.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.036+0000 7ff47f44f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.036+0000 7ff47f44f640 1 -- 172.21.15.69:0/3518222716 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff478142e40 con 0x7ff4780b2310 2024-01-21T05:05:51.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.037+0000 7ff47f44f640 10 client.21586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:51.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.037+0000 7ff47f44f640 10 monclient: _renew_subs 2024-01-21T05:05:51.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.037+0000 7ff47f44f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.037+0000 7ff47f44f640 1 -- 172.21.15.69:0/3518222716 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff478138480 con 0x7ff4780b2310 2024-01-21T05:05:51.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.037+0000 7ff47f44f640 1 librados: init done 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.038+0000 7ff46dffb640 1 -- 172.21.15.69:0/3518222716 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47000f040 con 0x7ff4780b2310 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.038+0000 7ff46dffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.038+0000 7ff46dffb640 1 -- 172.21.15.69:0/3518222716 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4700147e0 con 0x7ff4780b2310 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.038+0000 7ff46dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.038+0000 7ff46dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.038+0000 7ff46dffb640 10 monclient: dump: 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.043 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.043 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.038+0000 7ff46f7fe640 4 set_mon_vals no callback set 2024-01-21T05:05:51.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.038+0000 7ff47f44f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:51.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.039+0000 7ff47f44f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:51.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.039+0000 7ff47f44f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.039+0000 7ff47f44f640 1 -- 172.21.15.69:0/3518222716 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff478142b20 con 0x7ff4780b2310 2024-01-21T05:05:51.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.059+0000 7ff46dffb640 1 -- 172.21.15.69:0/3518222716 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7ff470014940 con 0x7ff4780b2310 2024-01-21T05:05:51.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.059+0000 7ff46dffb640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7ff45c06c860 0x7ff45c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.059+0000 7ff47d1eb640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7ff45c06c860 0x7ff45c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.059+0000 7ff47d1eb640 10 monclient: get_auth_request con 0x7ff45c06c860 auth_method 0 2024-01-21T05:05:51.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.060+0000 7ff46dffb640 10 client.21586.objecter ms_handle_connect 0x7ff45c06c860 2024-01-21T05:05:51.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.060+0000 7ff47d1eb640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7ff45c06c860 0x7ff45c06ed20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff4740016e0 tx=0x7ff474009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:51.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.289+0000 7ff46dffb640 1 -- 172.21.15.69:0/3518222716 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7ff470059510 con 0x7ff4780b2310 2024-01-21T05:05:51.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.289+0000 7ff46dffb640 10 client.21586.objecter ms_dispatch 0x7ff478000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:51.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.289+0000 7ff46dffb640 3 client.21586.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:51.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.289+0000 7ff46dffb640 3 client.21586.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:51.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.289+0000 7ff46dffb640 20 client.21586.objecter dump_active .. 0 homeless 2024-01-21T05:05:51.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.298+0000 7ff46dffb640 1 -- 172.21.15.69:0/3518222716 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7ff470055850 con 0x7ff4780b2310 2024-01-21T05:05:51.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.298+0000 7ff46dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:51.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.298+0000 7ff46dffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.472+0000 7ff47f44f640 1 -- 172.21.15.69:0/3518222716 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff478138c70 con 0x7ff45c06c860 2024-01-21T05:05:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.473+0000 7ff46dffb640 1 -- 172.21.15.69:0/3518222716 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff474001cc0 con 0x7ff45c06c860 2024-01-21T05:05:51.474 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 3 to deep-scrub 2024-01-21T05:05:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 10 librados: watch_flush enter 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 10 librados: watch_flush exit 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 20 client.21586.objecter shutdown clearing up homeless session... 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 10 client.21586.objecter successfully canceled tick 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 -- 172.21.15.69:0/3518222716 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7ff45c06c860 msgr2=0x7ff45c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7ff45c06c860 0x7ff45c06ed20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff4740016e0 tx=0x7ff474009040 comp rx=0 tx=0).stop 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 10 monclient: shutdown 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 -- 172.21.15.69:0/3518222716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4780b2310 msgr2=0x7ff4781407e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4780b2310 0x7ff4781407e0 secure :-1 s=READY pgs=5228 cs=0 l=1 rev1=1 crypto rx=0x7ff470009d90 tx=0x7ff47000c490 comp rx=0 tx=0).stop 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 -- 172.21.15.69:0/3518222716 shutdown_connections 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7ff45c06c860 0x7ff45c06ed20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4780b2310 0x7ff4781407e0 unknown :-1 s=CLOSED pgs=5228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7ff4780a9d10 0x7ff4781402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 --2- 172.21.15.69:0/3518222716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4780a8b20 0x7ff478137f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 -- 172.21.15.69:0/3518222716 >> 172.21.15.69:0/3518222716 conn(0x7ff478066fb0 msgr2=0x7ff478137200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:51.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 -- 172.21.15.69:0/3518222716 shutdown_connections 2024-01-21T05:05:51.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 -- 172.21.15.69:0/3518222716 wait complete. 2024-01-21T05:05:51.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.475+0000 7ff47f44f640 1 librados: shutdown 2024-01-21T05:05:51.488 INFO:tasks.ceph:Scrubbing osd.4 2024-01-21T05:05:51.488 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.593+0000 7fb601a46640 10 monclient: get_monmap_and_config 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.593+0000 7fb601a46640 10 monclient: build_initial_monmap 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.593+0000 7fb601a46640 10 monclient: monmap: 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:51.594315+0000 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:51.594315+0000 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:51.594 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:51.595 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:51.595 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 1 Processor -- start 2024-01-21T05:05:51.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 1 -- start start 2024-01-21T05:05:51.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 10 monclient: init 2024-01-21T05:05:51.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:51.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-01-21T05:05:51.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5fc0a8a40 0x7fb5fc145b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 10 monclient(hunting): picked mon.noname-a con 0x7fb5fc0a8a40 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:51.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb5fc0b2210 0x7fb5fc13f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 10 monclient(hunting): picked mon.noname-c con 0x7fb5fc0b2210 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:51.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5fc0a8e20 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:51.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc145700 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5fc0aa7b0 con 0x7fb5fc0a8a40 2024-01-21T05:05:51.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5fc0aa9b0 con 0x7fb5fc0b2210 2024-01-21T05:05:51.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:51.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.594+0000 7fb601a46640 10 monclient(hunting): authenticate will time out at 3309.936121s 2024-01-21T05:05:51.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.596+0000 7fb5f27fc640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7fb5fc0b2210 0x7fb5fc13f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.596+0000 7fb5fb7fe640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fb5fc0a8e20 0x7fb5fc141fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:39416/0 (socket says 172.21.15.69:39416) 2024-01-21T05:05:51.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.596+0000 7fb5fb7fe640 1 -- 172.21.15.69:0/4086433390 learned_addr learned my addr 172.21.15.69:0/4086433390 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T05:05:51.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.597+0000 7fb5f27fc640 10 monclient(hunting): get_auth_request con 0x7fb5fc0b2210 auth_method 0 2024-01-21T05:05:51.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.597+0000 7fb5f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.597+0000 7fb5f27fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.597+0000 7fb5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.597+0000 7fb5f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.597+0000 7fb5f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.597+0000 7fb5f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.599+0000 7fb5f27fc640 10 monclient(hunting): handle_auth_done global_id 21589 payload 274 2024-01-21T05:05:51.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.599+0000 7fb5f27fc640 1 -- 172.21.15.69:0/4086433390 >> v2:172.21.15.69:3300/0 conn(0x7fb5fc0a8a40 msgr2=0x7fb5fc145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:51.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.599+0000 7fb5f27fc640 1 --2- 172.21.15.69:0/4086433390 >> v2:172.21.15.69:3300/0 conn(0x7fb5fc0a8a40 0x7fb5fc145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.599+0000 7fb5f27fc640 1 -- 172.21.15.69:0/4086433390 >> v1:172.21.15.144:6789/0 conn(0x7fb5fc0a8e20 legacy=0x7fb5fc141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.599+0000 7fb5f27fc640 10 monclient: _finish_hunting 0 2024-01-21T05:05:51.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.599+0000 7fb5f27fc640 1 monclient: found mon.noname-c 2024-01-21T05:05:51.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.599+0000 7fb5f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:51.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.599+0000 7fb5f27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.599+0000 7fb5f27fc640 1 -- 172.21.15.69:0/4086433390 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc0aac90 con 0x7fb5fc0b2210 2024-01-21T05:05:51.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.649+0000 7fb5f27fc640 1 --2- 172.21.15.69:0/4086433390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5fc0b2210 0x7fb5fc13f1c0 secure :-1 s=READY pgs=5230 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0b1bb0 tx=0x7fb5e402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6ad163be06cf082 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:51.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 1 -- 172.21.15.69:0/4086433390 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4031830 con 0x7fb5fc0b2210 2024-01-21T05:05:51.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 10 monclient: dump: 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.657 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 10 monclient: _finish_auth 0 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 10 monclient: _check_auth_tickets 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 1 -- 172.21.15.69:0/4086433390 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e4031990 con 0x7fb5fc0b2210 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 1 -- 172.21.15.69:0/4086433390 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4031d20 con 0x7fb5fc0b2210 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb601a46640 5 monclient: authenticate success, global_id 21589 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb601a46640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb601a46640 4 set_mon_vals no callback set 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb5fa7fc640 10 monclient: dump: 2024-01-21T05:05:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.659 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb601a46640 10 monclient: shutdown 2024-01-21T05:05:51.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb601a46640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:51.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb601a46640 1 -- 172.21.15.69:0/4086433390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5fc0b2210 msgr2=0x7fb5fc13f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.656+0000 7fb601a46640 1 --2- 172.21.15.69:0/4086433390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5fc0b2210 0x7fb5fc13f1c0 secure :-1 s=READY pgs=5230 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0b1bb0 tx=0x7fb5e402b8d0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.657+0000 7fb601a46640 1 -- 172.21.15.69:0/4086433390 shutdown_connections 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.657+0000 7fb601a46640 1 --2- 172.21.15.69:0/4086433390 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5fc0b2210 0x7fb5fc13f1c0 unknown :-1 s=CLOSED pgs=5230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.657+0000 7fb601a46640 1 --2- 172.21.15.69:0/4086433390 >> v2:172.21.15.69:3300/0 conn(0x7fb5fc0a8a40 0x7fb5fc145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.657+0000 7fb601a46640 1 -- 172.21.15.69:0/4086433390 >> 172.21.15.69:0/4086433390 conn(0x7fb5fc066f10 msgr2=0x7fb5fc0b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.657+0000 7fb601a46640 1 -- 172.21.15.69:0/4086433390 shutdown_connections 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.657+0000 7fb601a46640 1 -- 172.21.15.69:0/4086433390 wait complete. 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.657+0000 7fb601a46640 10 monclient: build_initial_monmap 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.657+0000 7fb601a46640 10 monclient: monmap: 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.664 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.657+0000 7fb601a46640 1 librados: starting msgr at 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 1 librados: starting objecter 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 1 Processor -- start 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 1 -- start start 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 1 librados: setting wanted keys 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 1 librados: calling monclient init 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 10 monclient: init 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5fc0a8a40 0x7fb5fc143f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 10 monclient(hunting): picked mon.noname-a con 0x7fb5fc0a8a40 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5fc0a8e20 0x7fb5fc144520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 10 monclient(hunting): picked mon.noname-b con 0x7fb5fc0a8e20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5fc0b2210 0x7fb5fc1410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 10 monclient(hunting): picked mon.noname-c con 0x7fb5fc0b2210 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc144c80 con 0x7fb5fc0a8a40 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc140cb0 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc145110 con 0x7fb5fc0b2210 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.658+0000 7fb601a46640 10 monclient(hunting): authenticate will time out at 3310.000060s 2024-01-21T05:05:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5faffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5fc0a8a40 0x7fb5fc143f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5fb7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5fc0b2210 0x7fb5fc1410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5f27fc640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5fc0a8e20 0x7fb5fc144520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5fb7fe640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5fc0b2210 0x7fb5fc1410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:48652/0 (socket says 172.21.15.69:48652) 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5fb7fe640 1 -- 172.21.15.69:0/2126555601 learned_addr learned my addr 172.21.15.69:0/2126555601 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5fb7fe640 10 monclient(hunting): get_auth_request con 0x7fb5fc0b2210 auth_method 0 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5fb7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5f27fc640 10 monclient(hunting): get_auth_request con 0x7fb5fc0a8e20 auth_method 0 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5f27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5f27fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5faffd640 10 monclient(hunting): get_auth_request con 0x7fb5fc0a8a40 auth_method 0 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5faffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.659+0000 7fb5faffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.660+0000 7fb5f27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.660+0000 7fb5f27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.660+0000 7fb5f27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.660+0000 7fb5fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.660+0000 7fb5fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.660+0000 7fb5fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.660+0000 7fb5faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.660+0000 7fb5faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.660+0000 7fb5faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.662+0000 7fb5f27fc640 10 monclient(hunting): handle_auth_done global_id 21592 payload 824 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.662+0000 7fb5f27fc640 1 -- 172.21.15.69:0/2126555601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5fc0b2210 msgr2=0x7fb5fc1410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.662+0000 7fb5f27fc640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fb5fc0b2210 0x7fb5fc1410d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.662+0000 7fb5f27fc640 1 -- 172.21.15.69:0/2126555601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5fc0a8a40 msgr2=0x7fb5fc143f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.662+0000 7fb5f27fc640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5fc0a8a40 0x7fb5fc143f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.662+0000 7fb5f27fc640 10 monclient: _finish_hunting 0 2024-01-21T05:05:51.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.662+0000 7fb5f27fc640 1 monclient: found mon.noname-b 2024-01-21T05:05:51.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.662+0000 7fb5f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:51.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.662+0000 7fb5f27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.662+0000 7fb5f27fc640 1 -- 172.21.15.69:0/2126555601 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4009400 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.665+0000 7fb5f27fc640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5fc0a8e20 0x7fb5fc144520 secure :-1 s=READY pgs=5231 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4033000 tx=0x7fb5e40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.665+0000 7fb5f8ff9640 10 client.?.objecter ms_handle_connect 0x7fb5fc0a8e20 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.665+0000 7fb5f8ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 1 -- 172.21.15.69:0/2126555601 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e403ba70 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 10 monclient: dump: 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.671 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.672 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 10 monclient: _finish_auth 0 2024-01-21T05:05:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 10 monclient: _check_auth_tickets 2024-01-21T05:05:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 1 -- 172.21.15.69:0/2126555601 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e403bbd0 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 1 -- 172.21.15.69:0/2126555601 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40453f0 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f8ff9640 10 monclient: dump: 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb601a46640 5 monclient: authenticate success, global_id 21592 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb601a46640 10 monclient: _renew_subs 2024-01-21T05:05:51.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb5f9ffb640 4 set_mon_vals no callback set 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb601a46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb601a46640 1 -- 172.21.15.69:0/2126555601 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5fc13fde0 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb601a46640 10 client.21592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb601a46640 10 monclient: _renew_subs 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb601a46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb601a46640 1 -- 172.21.15.69:0/2126555601 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5fc1417f0 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.668+0000 7fb601a46640 1 librados: init done 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.669+0000 7fb601a46640 10 client.21592.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.669+0000 7fb601a46640 15 client.21592.objecter _session_command_op_assign -1 1 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.669+0000 7fb601a46640 20 client.21592.objecter _get_session osd=-1 returning homeless 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.669+0000 7fb601a46640 10 client.21592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.669+0000 7fb601a46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.669+0000 7fb601a46640 1 -- 172.21.15.69:0/2126555601 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb5fc04e970 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.687+0000 7fb5f8ff9640 1 -- 172.21.15.69:0/2126555601 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fb5e404e450 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.687+0000 7fb5f8ff9640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fb5dc06c8b0 0x7fb5dc06ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.688+0000 7fb5faffd640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fb5dc06c8b0 0x7fb5dc06ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.688+0000 7fb5faffd640 10 monclient: get_auth_request con 0x7fb5dc06c8b0 auth_method 0 2024-01-21T05:05:51.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.688+0000 7fb5faffd640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fb5dc06c8b0 0x7fb5dc06ed70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009990 tx=0x7fb5ec009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:51.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.688+0000 7fb5f8ff9640 10 client.21592.objecter ms_handle_connect 0x7fb5dc06c8b0 2024-01-21T05:05:51.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 1 -- 172.21.15.69:0/2126555601 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7fb5e407afd0 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 10 client.21592.objecter ms_dispatch 0x7fb5fc000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 3 client.21592.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 3 client.21592.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 10 client.21592.objecter checking command 1 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] conn(0x7fb5dc0745a0 0x7fb5dc076a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 20 client.21592.objecter _get_session s=0x7fb5dc073ed0 osd=4 3 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 20 client.21592.objecter put_session s=0x7fb5dc073ed0 osd=4 3 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 15 client.21592.objecter _session_command_op_remove -1 1 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 20 client.21592.objecter _get_session s=0x7fb5dc073ed0 osd=4 3 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 20 client.21592.objecter get_session s=0x7fb5dc073ed0 osd=4 3 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 15 client.21592.objecter _session_command_op_assign 4 1 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 20 client.21592.objecter put_session s=0x7fb5dc073ed0 osd=4 4 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 10 client.21592.objecter _send_command 1 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 1 -- 172.21.15.69:0/2126555601 --> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5dc077110 con 0x7fb5dc0745a0 2024-01-21T05:05:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.689+0000 7fb5f8ff9640 20 client.21592.objecter dump_active .. 0 homeless 2024-01-21T05:05:51.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.690+0000 7fb5fb7fe640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] conn(0x7fb5dc0745a0 0x7fb5dc076a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.690+0000 7fb5fb7fe640 10 monclient: get_auth_request con 0x7fb5dc0745a0 auth_method 0 2024-01-21T05:05:51.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.690+0000 7fb5f8ff9640 1 -- 172.21.15.69:0/2126555601 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=1109) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5e4076c10 con 0x7fb5fc0a8e20 2024-01-21T05:05:51.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.690+0000 7fb5f8ff9640 10 monclient: handle_get_version_reply finishing 1 version 1109 2024-01-21T05:05:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.693+0000 7fb5fb7fe640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] conn(0x7fb5dc0745a0 0x7fb5dc076a40 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.693+0000 7fb5f8ff9640 10 client.21592.objecter ms_handle_connect 0x7fb5dc0745a0 2024-01-21T05:05:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.717+0000 7fb5f8ff9640 1 -- 172.21.15.69:0/2126555601 <== osd.4 v2:172.21.15.144:6826/3897583001 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb5dc077110 con 0x7fb5dc0745a0 2024-01-21T05:05:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.717+0000 7fb5f8ff9640 10 client.21592.objecter ms_dispatch 0x7fb5fc000b90 command_reply(tid 1: 0 ) v1 2024-01-21T05:05:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.717+0000 7fb5f8ff9640 10 client.21592.objecter _finish_command 1 = system:0 2024-01-21T05:05:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.717+0000 7fb5f8ff9640 20 client.21592.objecter put_session s=0x7fb5dc073ed0 osd=4 4 2024-01-21T05:05:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.717+0000 7fb5f8ff9640 15 client.21592.objecter _session_command_op_remove 4 1 2024-01-21T05:05:51.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.730+0000 7fb601a46640 10 client.21592.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T05:05:51.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.730+0000 7fb601a46640 15 client.21592.objecter _session_command_op_assign -1 2 2024-01-21T05:05:51.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.730+0000 7fb601a46640 20 client.21592.objecter _get_session s=0x7fb5dc073ed0 osd=4 3 2024-01-21T05:05:51.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.730+0000 7fb601a46640 20 client.21592.objecter put_session s=0x7fb5dc073ed0 osd=4 3 2024-01-21T05:05:51.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.730+0000 7fb601a46640 20 client.21592.objecter _get_session s=0x7fb5dc073ed0 osd=4 3 2024-01-21T05:05:51.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.730+0000 7fb601a46640 15 client.21592.objecter _session_command_op_remove -1 2 2024-01-21T05:05:51.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.730+0000 7fb601a46640 20 client.21592.objecter get_session s=0x7fb5dc073ed0 osd=4 3 2024-01-21T05:05:51.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.730+0000 7fb601a46640 15 client.21592.objecter _session_command_op_assign 4 2 2024-01-21T05:05:51.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.730+0000 7fb601a46640 20 client.21592.objecter put_session s=0x7fb5dc073ed0 osd=4 4 2024-01-21T05:05:51.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.730+0000 7fb601a46640 10 client.21592.objecter _send_command 2 2024-01-21T05:05:51.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.730+0000 7fb601a46640 1 -- 172.21.15.69:0/2126555601 --> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb5fc144dc0 con 0x7fb5dc0745a0 2024-01-21T05:05:51.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb5f8ff9640 1 -- 172.21.15.69:0/2126555601 <== osd.4 v2:172.21.15.144:6826/3897583001 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb5fc144dc0 con 0x7fb5dc0745a0 2024-01-21T05:05:51.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb5f8ff9640 10 client.21592.objecter ms_dispatch 0x7fb5fc000b90 command_reply(tid 2: 0 ) v1 2024-01-21T05:05:51.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb5f8ff9640 10 client.21592.objecter _finish_command 2 = system:0 2024-01-21T05:05:51.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb5f8ff9640 20 client.21592.objecter put_session s=0x7fb5dc073ed0 osd=4 4 2024-01-21T05:05:51.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb5f8ff9640 15 client.21592.objecter _session_command_op_remove 4 2 2024-01-21T05:05:51.745 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-21T05:05:51.745 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T05:05:51.745 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-21T05:05:51.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb601a46640 10 librados: watch_flush enter 2024-01-21T05:05:51.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb601a46640 10 librados: watch_flush exit 2024-01-21T05:05:51.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb601a46640 10 client.21592.objecter close_session for osd.4 2024-01-21T05:05:51.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb601a46640 1 -- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] conn(0x7fb5dc0745a0 msgr2=0x7fb5dc076a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb601a46640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:6826/3897583001,v1:172.21.15.144:6827/3897583001] conn(0x7fb5dc0745a0 0x7fb5dc076a40 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb601a46640 20 client.21592.objecter put_session s=0x7fb5dc073ed0 osd=4 1 2024-01-21T05:05:51.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb601a46640 20 client.21592.objecter shutdown clearing up homeless session... 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.743+0000 7fb601a46640 10 client.21592.objecter successfully canceled tick 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb601a46640 1 -- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fb5dc06c8b0 msgr2=0x7fb5dc06ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb601a46640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fb5dc06c8b0 0x7fb5dc06ed70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009990 tx=0x7fb5ec009110 comp rx=0 tx=0).stop 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb601a46640 10 monclient: shutdown 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb601a46640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb601a46640 1 -- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5fc0a8e20 msgr2=0x7fb5fc144520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb601a46640 1 --2- 172.21.15.69:0/2126555601 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb5fc0a8e20 0x7fb5fc144520 secure :-1 s=READY pgs=5231 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4033000 tx=0x7fb5e40046c0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb5fb7fe640 1 -- 172.21.15.69:0/2126555601 reap_dead start 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb601a46640 1 -- 172.21.15.69:0/2126555601 shutdown_connections 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb601a46640 1 -- 172.21.15.69:0/2126555601 >> 172.21.15.69:0/2126555601 conn(0x7fb5fc066f10 msgr2=0x7fb5fc0b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb601a46640 1 -- 172.21.15.69:0/2126555601 shutdown_connections 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb601a46640 1 -- 172.21.15.69:0/2126555601 wait complete. 2024-01-21T05:05:51.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.744+0000 7fb601a46640 1 librados: shutdown 2024-01-21T05:05:51.755 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-21T05:05:51.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.854+0000 7f678d34e640 10 monclient: get_monmap_and_config 2024-01-21T05:05:51.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.854+0000 7f678d34e640 10 monclient: build_initial_monmap 2024-01-21T05:05:51.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.854+0000 7f678d34e640 10 monclient: monmap: 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:51.855897+0000 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:51.855897+0000 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:51.856 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.855+0000 7f678d34e640 1 Processor -- start 2024-01-21T05:05:51.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.855+0000 7f678d34e640 1 -- start start 2024-01-21T05:05:51.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.855+0000 7f678d34e640 10 monclient: init 2024-01-21T05:05:51.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:51.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-01-21T05:05:51.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f67880a8ab0 0x7f6788142640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 10 monclient(hunting): picked mon.noname-c con 0x7f67880a8ab0 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:51.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67880b2310 0x7f6788142b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 10 monclient(hunting): picked mon.noname-b con 0x7f67880b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:51.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67880a8e90 0x7f678813f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 10 monclient(hunting): picked mon.noname-a con 0x7f67880a8e90 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67880aa7d0 con 0x7f67880a8e90 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67880aaa20 con 0x7f67880a8ab0 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67880aad00 con 0x7f67880b2310 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f678d34e640 10 monclient(hunting): authenticate will time out at 3310.197380s 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f67867fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67880b2310 0x7f6788142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f6786ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f67880a8ab0 0x7f6788142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.856+0000 7f67877fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67880a8e90 0x7f678813f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f6786ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f67880a8ab0 0x7f6788142640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:41378/0 (socket says 172.21.15.69:41378) 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f6786ffd640 1 -- 172.21.15.69:0/979669125 learned_addr learned my addr 172.21.15.69:0/979669125 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f6786ffd640 10 monclient(hunting): get_auth_request con 0x7f67880a8ab0 auth_method 0 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67877fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67880a8e90 0x7f678813f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41728/0 (socket says 172.21.15.69:41728) 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f6786ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f6786ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f6786ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67877fe640 10 monclient(hunting): get_auth_request con 0x7f67880a8e90 auth_method 0 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67877fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67877fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67867fc640 10 monclient(hunting): get_auth_request con 0x7f67880b2310 auth_method 0 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67867fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67867fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f6786ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f6786ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f6786ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.857+0000 7f67877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.859+0000 7f6786ffd640 10 monclient(hunting): handle_auth_done global_id 21595 payload 274 2024-01-21T05:05:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.859+0000 7f6786ffd640 1 -- 172.21.15.69:0/979669125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67880b2310 msgr2=0x7f6788142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.859+0000 7f6786ffd640 1 --2- 172.21.15.69:0/979669125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67880b2310 0x7f6788142b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.859+0000 7f6786ffd640 1 -- 172.21.15.69:0/979669125 >> v2:172.21.15.69:3300/0 conn(0x7f67880a8e90 msgr2=0x7f678813f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.859+0000 7f6786ffd640 1 --2- 172.21.15.69:0/979669125 >> v2:172.21.15.69:3300/0 conn(0x7f67880a8e90 0x7f678813f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.859+0000 7f6786ffd640 10 monclient: _finish_hunting 0 2024-01-21T05:05:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.859+0000 7f6786ffd640 1 monclient: found mon.noname-c 2024-01-21T05:05:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.859+0000 7f6786ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:51.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.859+0000 7f6786ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.859+0000 7f6786ffd640 1 -- 172.21.15.69:0/979669125 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67880aafe0 con 0x7f67880a8ab0 2024-01-21T05:05:51.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.861+0000 7f6786ffd640 1 --2- 172.21.15.69:0/979669125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67880a8ab0 0x7f6788142640 secure :-1 s=READY pgs=5232 cs=0 l=1 rev1=1 crypto rx=0x7f6778009870 tx=0x7f677802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35356beab7ab2ccf server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:51.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 1 -- 172.21.15.69:0/979669125 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6778031640 con 0x7f67880a8ab0 2024-01-21T05:05:51.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:05:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 10 monclient: dump: 2024-01-21T05:05:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.866 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.867 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.868 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.868 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.868 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.868 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 10 monclient: _finish_auth 0 2024-01-21T05:05:51.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 10 monclient: _check_auth_tickets 2024-01-21T05:05:51.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:51.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 1 -- 172.21.15.69:0/979669125 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67780317a0 con 0x7f67880a8ab0 2024-01-21T05:05:51.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f678d34e640 5 monclient: authenticate success, global_id 21595 2024-01-21T05:05:51.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f678d34e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f678d34e640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 1 -- 172.21.15.69:0/979669125 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6778031b30 con 0x7f67880a8ab0 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f678d34e640 4 set_mon_vals no callback set 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.864+0000 7f6785ffb640 10 monclient: dump: 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.869 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.870 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.870 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.870 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.865+0000 7f678d34e640 10 monclient: shutdown 2024-01-21T05:05:51.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.865+0000 7f678d34e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:51.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.865+0000 7f678d34e640 1 -- 172.21.15.69:0/979669125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67880a8ab0 msgr2=0x7f6788142640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.865+0000 7f678d34e640 1 --2- 172.21.15.69:0/979669125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67880a8ab0 0x7f6788142640 secure :-1 s=READY pgs=5232 cs=0 l=1 rev1=1 crypto rx=0x7f6778009870 tx=0x7f677802b8f0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.865+0000 7f678d34e640 1 -- 172.21.15.69:0/979669125 shutdown_connections 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.865+0000 7f678d34e640 1 --2- 172.21.15.69:0/979669125 >> v2:172.21.15.69:3300/0 conn(0x7f67880a8e90 0x7f678813f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.865+0000 7f678d34e640 1 --2- 172.21.15.69:0/979669125 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67880b2310 0x7f6788142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.865+0000 7f678d34e640 1 --2- 172.21.15.69:0/979669125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67880a8ab0 0x7f6788142640 unknown :-1 s=CLOSED pgs=5232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.865+0000 7f678d34e640 1 -- 172.21.15.69:0/979669125 >> 172.21.15.69:0/979669125 conn(0x7f6788066f60 msgr2=0x7f67880b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.865+0000 7f678d34e640 1 -- 172.21.15.69:0/979669125 shutdown_connections 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.865+0000 7f678d34e640 1 -- 172.21.15.69:0/979669125 wait complete. 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.866+0000 7f678d34e640 10 monclient: build_initial_monmap 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.866+0000 7f678d34e640 10 monclient: monmap: 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:51.874 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.866+0000 7f678d34e640 1 librados: starting msgr at 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.866+0000 7f678d34e640 1 librados: starting objecter 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.866+0000 7f678d34e640 1 Processor -- start 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.866+0000 7f678d34e640 1 -- start start 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.866+0000 7f678d34e640 1 librados: setting wanted keys 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.866+0000 7f678d34e640 1 librados: calling monclient init 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.866+0000 7f678d34e640 10 monclient: init 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T05:05:51.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67880a8ab0 0x7f67880b67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 10 monclient(hunting): picked mon.noname-a con 0x7f67880a8ab0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67880a8e90 0x7f67881da4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 10 monclient(hunting): picked mon.noname-c con 0x7f67880a8e90 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67880b2310 0x7f67880b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 10 monclient(hunting): picked mon.noname-b con 0x7f67880b2310 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6788145620 con 0x7f67880a8ab0 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6788145190 con 0x7f67880b2310 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6788145350 con 0x7f67880a8e90 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:51.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f678d34e640 10 monclient(hunting): authenticate will time out at 3310.208663s 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f67867fc640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67880a8e90 0x7f67881da4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.867+0000 7f6786ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67880a8ab0 0x7f67880b67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67877fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67880b2310 0x7f67880b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67877fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67880b2310 0x7f67880b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:41392/0 (socket says 172.21.15.69:41392) 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f6786ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67880a8ab0 0x7f67880b67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41734/0 (socket says 172.21.15.69:41734) 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67877fe640 1 -- 172.21.15.69:0/2280895370 learned_addr learned my addr 172.21.15.69:0/2280895370 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67877fe640 10 monclient(hunting): get_auth_request con 0x7f67880b2310 auth_method 0 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67877fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67877fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67867fc640 10 monclient(hunting): get_auth_request con 0x7f67880a8e90 auth_method 0 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67867fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67867fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f6786ffd640 10 monclient(hunting): get_auth_request con 0x7f67880a8ab0 auth_method 0 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f6786ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f6786ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f6786ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f6786ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f6786ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f6786ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.868+0000 7f67867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:51.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.871+0000 7f67877fe640 10 monclient(hunting): handle_auth_done global_id 21598 payload 824 2024-01-21T05:05:51.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.871+0000 7f67877fe640 1 -- 172.21.15.69:0/2280895370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67880a8e90 msgr2=0x7f67881da4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.871+0000 7f67877fe640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67880a8e90 0x7f67881da4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.871+0000 7f67877fe640 1 -- 172.21.15.69:0/2280895370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67880a8ab0 msgr2=0x7f67880b67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:51.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.871+0000 7f67877fe640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67880a8ab0 0x7f67880b67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:51.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.871+0000 7f67877fe640 10 monclient: _finish_hunting 0 2024-01-21T05:05:51.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.871+0000 7f67877fe640 1 monclient: found mon.noname-b 2024-01-21T05:05:51.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.871+0000 7f67877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:51.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.871+0000 7f67877fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.871+0000 7f67877fe640 1 -- 172.21.15.69:0/2280895370 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67780094d0 con 0x7f67880b2310 2024-01-21T05:05:51.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.873+0000 7f67877fe640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67880b2310 0x7f67880b0540 secure :-1 s=READY pgs=5233 cs=0 l=1 rev1=1 crypto rx=0x7f678000efc0 tx=0x7f678000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:51.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.873+0000 7f676f7fe640 10 client.?.objecter ms_handle_connect 0x7f67880b2310 2024-01-21T05:05:51.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.873+0000 7f676f7fe640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:51.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 1 -- 172.21.15.69:0/2280895370 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6780009ca0 con 0x7f67880b2310 2024-01-21T05:05:51.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:51.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 10 monclient: dump: 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 10 monclient: _finish_auth 0 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 10 monclient: _check_auth_tickets 2024-01-21T05:05:51.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 1 -- 172.21.15.69:0/2280895370 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678000f040 con 0x7f67880b2310 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 1 -- 172.21.15.69:0/2280895370 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67800146c0 con 0x7f67880b2310 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f678d34e640 5 monclient: authenticate success, global_id 21598 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f678d34e640 10 monclient: _renew_subs 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f678d34e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f678d34e640 1 -- 172.21.15.69:0/2280895370 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6788144cf0 con 0x7f67880b2310 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f676f7fe640 10 monclient: dump: 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:51.885 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f67857fa640 4 set_mon_vals no callback set 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f678d34e640 10 client.21598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f678d34e640 10 monclient: _renew_subs 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f678d34e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f678d34e640 1 -- 172.21.15.69:0/2280895370 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6788145d20 con 0x7f67880b2310 2024-01-21T05:05:51.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.878+0000 7f678d34e640 1 librados: init done 2024-01-21T05:05:51.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.879+0000 7f678d34e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:51.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.879+0000 7f678d34e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:51.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.879+0000 7f678d34e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:51.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:51.879+0000 7f678d34e640 1 -- 172.21.15.69:0/2280895370 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678804e970 con 0x7f67880b2310 2024-01-21T05:05:52.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.129+0000 7f676f7fe640 1 -- 172.21.15.69:0/2280895370 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f6780014820 con 0x7f67880b2310 2024-01-21T05:05:52.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.130+0000 7f676f7fe640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f677406c8b0 0x7f677406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.130+0000 7f6786ffd640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f677406c8b0 0x7f677406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:52.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.130+0000 7f6786ffd640 10 monclient: get_auth_request con 0x7f677406c8b0 auth_method 0 2024-01-21T05:05:52.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.130+0000 7f6786ffd640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f677406c8b0 0x7f677406ed70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6778036000 tx=0x7f677803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:52.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.130+0000 7f676f7fe640 10 client.21598.objecter ms_handle_connect 0x7f677406c8b0 2024-01-21T05:05:52.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.132+0000 7f676f7fe640 1 -- 172.21.15.69:0/2280895370 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f6780059730 con 0x7f67880b2310 2024-01-21T05:05:52.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.132+0000 7f676f7fe640 10 client.21598.objecter ms_dispatch 0x7f6788000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:52.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.132+0000 7f676f7fe640 3 client.21598.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:52.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.132+0000 7f676f7fe640 3 client.21598.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.132+0000 7f676f7fe640 20 client.21598.objecter dump_active .. 0 homeless 2024-01-21T05:05:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.144+0000 7f676f7fe640 1 -- 172.21.15.69:0/2280895370 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6780055a70 con 0x7f67880b2310 2024-01-21T05:05:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.144+0000 7f676f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:52.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.144+0000 7f676f7fe640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:52.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.320+0000 7f678d34e640 1 -- 172.21.15.69:0/2280895370 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6788072860 con 0x7f677406c8b0 2024-01-21T05:05:52.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.320+0000 7f676f7fe640 1 -- 172.21.15.69:0/2280895370 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6788072860 con 0x7f677406c8b0 2024-01-21T05:05:52.321 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 4 to deep-scrub 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 10 librados: watch_flush enter 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 10 librados: watch_flush exit 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 20 client.21598.objecter shutdown clearing up homeless session... 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 10 client.21598.objecter successfully canceled tick 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 -- 172.21.15.69:0/2280895370 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f677406c8b0 msgr2=0x7f677406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f677406c8b0 0x7f677406ed70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6778036000 tx=0x7f677803c040 comp rx=0 tx=0).stop 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 10 monclient: shutdown 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 -- 172.21.15.69:0/2280895370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67880b2310 msgr2=0x7f67880b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67880b2310 0x7f67880b0540 secure :-1 s=READY pgs=5233 cs=0 l=1 rev1=1 crypto rx=0x7f678000efc0 tx=0x7f678000c490 comp rx=0 tx=0).stop 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 -- 172.21.15.69:0/2280895370 shutdown_connections 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f677406c8b0 0x7f677406ed70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67880b2310 0x7f67880b0540 unknown :-1 s=CLOSED pgs=5233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f67880a8e90 0x7f67881da4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 --2- 172.21.15.69:0/2280895370 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67880a8ab0 0x7f67880b67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 -- 172.21.15.69:0/2280895370 >> 172.21.15.69:0/2280895370 conn(0x7f6788066f60 msgr2=0x7f6788142500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:52.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 -- 172.21.15.69:0/2280895370 shutdown_connections 2024-01-21T05:05:52.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 -- 172.21.15.69:0/2280895370 wait complete. 2024-01-21T05:05:52.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.322+0000 7f678d34e640 1 librados: shutdown 2024-01-21T05:05:52.334 INFO:tasks.ceph:Scrubbing osd.5 2024-01-21T05:05:52.334 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-21T05:05:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.436+0000 7fc5623d0640 10 monclient: get_monmap_and_config 2024-01-21T05:05:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.436+0000 7fc5623d0640 10 monclient: build_initial_monmap 2024-01-21T05:05:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.436+0000 7fc5623d0640 10 monclient: monmap: 2024-01-21T05:05:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:52.437575+0000 2024-01-21T05:05:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:52.437575+0000 2024-01-21T05:05:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.437 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:52.438 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:52.438 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 1 Processor -- start 2024-01-21T05:05:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 1 -- start start 2024-01-21T05:05:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 10 monclient: init 2024-01-21T05:05:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-01-21T05:05:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc55c066620 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc55c0aaed0 0x7fc55c071b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 10 monclient(hunting): picked mon.noname-a con 0x7fc55c0aaed0 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc55c146230 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c0aa7d0 con 0x7fc55c146230 2024-01-21T05:05:52.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c145d10 con 0x7fc55c066620 2024-01-21T05:05:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc55c145b20 con 0x7fc55c0aaed0 2024-01-21T05:05:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.437+0000 7fc5623d0640 10 monclient(hunting): authenticate will time out at 3310.779169s 2024-01-21T05:05:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.438+0000 7fc55b7fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc55c0aaed0 0x7fc55c071b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.438+0000 7fc55bfff640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7fc55c066620 0x7fc55c0a8910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:39430/0 (socket says 172.21.15.69:39430) 2024-01-21T05:05:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.438+0000 7fc55bfff640 1 -- 172.21.15.69:0/2083124629 learned_addr learned my addr 172.21.15.69:0/2083124629 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T05:05:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.438+0000 7fc55b7fe640 10 monclient(hunting): get_auth_request con 0x7fc55c0aaed0 auth_method 0 2024-01-21T05:05:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.438+0000 7fc55b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:52.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.438+0000 7fc55b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:52.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.438+0000 7fc55b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:52.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.439+0000 7fc55b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:52.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.439+0000 7fc55b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:52.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.439+0000 7fc55b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55affd640 1 -- 172.21.15.69:0/2083124629 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203679825 0 0) 0x7fc55c145d10 con 0x7fc55c066620 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55affd640 10 monclient(hunting): _negotiate 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55affd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55affd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55affd640 10 monclient(hunting): my global_id is 21601 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55affd640 1 -- 172.21.15.69:0/2083124629 --> v1:172.21.15.144:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc544003520 con 0x7fc55c066620 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55b7fe640 10 monclient(hunting): handle_auth_done global_id 17487 payload 274 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55b7fe640 1 -- 172.21.15.69:0/2083124629 >> v1:172.21.15.144:6789/0 conn(0x7fc55c066620 legacy=0x7fc55c0a8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55b7fe640 1 -- 172.21.15.69:0/2083124629 >> v1:172.21.15.69:6789/0 conn(0x7fc55c146230 legacy=0x7fc55c0720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55b7fe640 10 monclient: _finish_hunting 0 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55b7fe640 1 monclient: found mon.noname-a 2024-01-21T05:05:52.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.441+0000 7fc55b7fe640 1 -- 172.21.15.69:0/2083124629 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c0a9020 con 0x7fc55c0aaed0 2024-01-21T05:05:52.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.443+0000 7fc55b7fe640 1 --2- 172.21.15.69:0/2083124629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c0aaed0 0x7fc55c071b60 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7fc5500097a0 tx=0x7fc55002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e7278c5c117108 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:52.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 1 -- 172.21.15.69:0/2083124629 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55002e070 con 0x7fc55c0aaed0 2024-01-21T05:05:52.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:52.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:52.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 10 monclient: dump: 2024-01-21T05:05:52.448 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:52.448 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:52.448 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.448 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.448 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:52.448 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 10 monclient: _finish_auth 0 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 10 monclient: _check_auth_tickets 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 1 -- 172.21.15.69:0/2083124629 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5500315a0 con 0x7fc55c0aaed0 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc5623d0640 5 monclient: authenticate success, global_id 17487 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc5623d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 1 -- 172.21.15.69:0/2083124629 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550031930 con 0x7fc55c0aaed0 2024-01-21T05:05:52.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc55affd640 10 monclient: dump: 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc5623d0640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:52.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.447+0000 7fc5623d0640 4 set_mon_vals no callback set 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.448+0000 7fc5623d0640 10 monclient: shutdown 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.448+0000 7fc5623d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.448+0000 7fc5623d0640 1 -- 172.21.15.69:0/2083124629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c0aaed0 msgr2=0x7fc55c071b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.448+0000 7fc5623d0640 1 --2- 172.21.15.69:0/2083124629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c0aaed0 0x7fc55c071b60 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7fc5500097a0 tx=0x7fc55002b840 comp rx=0 tx=0).stop 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.448+0000 7fc5623d0640 1 -- 172.21.15.69:0/2083124629 shutdown_connections 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.448+0000 7fc5623d0640 1 --2- 172.21.15.69:0/2083124629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c0aaed0 0x7fc55c071b60 unknown :-1 s=CLOSED pgs=3499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.448+0000 7fc5623d0640 1 -- 172.21.15.69:0/2083124629 >> 172.21.15.69:0/2083124629 conn(0x7fc55c06a630 msgr2=0x7fc55c06aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.448+0000 7fc5623d0640 1 -- 172.21.15.69:0/2083124629 shutdown_connections 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.448+0000 7fc5623d0640 1 -- 172.21.15.69:0/2083124629 wait complete. 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.448+0000 7fc5623d0640 10 monclient: build_initial_monmap 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.448+0000 7fc5623d0640 10 monclient: monmap: 2024-01-21T05:05:52.454 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.449+0000 7fc5623d0640 1 librados: starting msgr at 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.449+0000 7fc5623d0640 1 librados: starting objecter 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.449+0000 7fc5623d0640 1 Processor -- start 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.449+0000 7fc5623d0640 1 -- start start 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.449+0000 7fc5623d0640 1 librados: setting wanted keys 2024-01-21T05:05:52.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.449+0000 7fc5623d0640 1 librados: calling monclient init 2024-01-21T05:05:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.449+0000 7fc5623d0640 10 monclient: init 2024-01-21T05:05:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.449+0000 7fc5623d0640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.449+0000 7fc5623d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T05:05:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.449+0000 7fc5623d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c066620 0x7fc55c13be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 10 monclient(hunting): picked mon.noname-a con 0x7fc55c066620 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc55c0aaed0 0x7fc55c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 10 monclient(hunting): picked mon.noname-b con 0x7fc55c0aaed0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc55c146230 0x7fc55c06f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 10 monclient(hunting): picked mon.noname-c con 0x7fc55c146230 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:52.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c1450f0 con 0x7fc55c066620 2024-01-21T05:05:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c06f5b0 con 0x7fc55c0aaed0 2024-01-21T05:05:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c070150 con 0x7fc55c146230 2024-01-21T05:05:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc5623d0640 10 monclient(hunting): authenticate will time out at 3310.791250s 2024-01-21T05:05:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc55c0aaed0 0x7fc55c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c066620 0x7fc55c13be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:52.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc56096d640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc55c146230 0x7fc55c06f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55b7fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc55c0aaed0 0x7fc55c06eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:41408/0 (socket says 172.21.15.69:41408) 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55b7fe640 1 -- 172.21.15.69:0/4105775790 learned_addr learned my addr 172.21.15.69:0/4105775790 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55b7fe640 10 monclient(hunting): get_auth_request con 0x7fc55c0aaed0 auth_method 0 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55b7fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55bfff640 10 monclient(hunting): get_auth_request con 0x7fc55c066620 auth_method 0 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55bfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.450+0000 7fc55bfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc56096d640 10 monclient(hunting): get_auth_request con 0x7fc55c146230 auth_method 0 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc56096d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc56096d640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:52.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc56096d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc55b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc55b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc55b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc55bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc55bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc55bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc56096d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc56096d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:52.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.451+0000 7fc56096d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:52.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.453+0000 7fc55b7fe640 10 monclient(hunting): handle_auth_done global_id 21604 payload 824 2024-01-21T05:05:52.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.453+0000 7fc55b7fe640 1 -- 172.21.15.69:0/4105775790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc55c146230 msgr2=0x7fc55c06f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.453+0000 7fc55b7fe640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7fc55c146230 0x7fc55c06f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.453+0000 7fc55b7fe640 1 -- 172.21.15.69:0/4105775790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c066620 msgr2=0x7fc55c13be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.453+0000 7fc55b7fe640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc55c066620 0x7fc55c13be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.454+0000 7fc55b7fe640 10 monclient: _finish_hunting 0 2024-01-21T05:05:52.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.454+0000 7fc55b7fe640 1 monclient: found mon.noname-b 2024-01-21T05:05:52.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.454+0000 7fc55b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:52.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.454+0000 7fc55b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:05:52.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.454+0000 7fc55b7fe640 1 -- 172.21.15.69:0/4105775790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc550009400 con 0x7fc55c0aaed0 2024-01-21T05:05:52.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.456+0000 7fc55b7fe640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc55c0aaed0 0x7fc55c06eb30 secure :-1 s=READY pgs=5235 cs=0 l=1 rev1=1 crypto rx=0x7fc5500098c0 tx=0x7fc5500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.456+0000 7fc558ff9640 10 client.?.objecter ms_handle_connect 0x7fc55c0aaed0 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.456+0000 7fc558ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc558ff9640 1 -- 172.21.15.69:0/4105775790 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550032840 con 0x7fc55c0aaed0 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc558ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc558ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc558ff9640 10 monclient: dump: 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:52.462 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:52.463 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:52.463 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc558ff9640 10 monclient: _finish_auth 0 2024-01-21T05:05:52.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc558ff9640 10 monclient: _check_auth_tickets 2024-01-21T05:05:52.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc558ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:52.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc558ff9640 1 -- 172.21.15.69:0/4105775790 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5500329a0 con 0x7fc55c0aaed0 2024-01-21T05:05:52.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc5623d0640 5 monclient: authenticate success, global_id 21604 2024-01-21T05:05:52.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc558ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:52.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc5623d0640 10 monclient: _renew_subs 2024-01-21T05:05:52.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc558ff9640 1 -- 172.21.15.69:0/4105775790 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550032d50 con 0x7fc55c0aaed0 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc5623d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.459+0000 7fc5623d0640 1 -- 172.21.15.69:0/4105775790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55c070430 con 0x7fc55c0aaed0 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc558ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc558ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc558ff9640 10 monclient: dump: 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.464 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:52.465 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:52.465 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:52.465 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc55a7fc640 4 set_mon_vals no callback set 2024-01-21T05:05:52.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc5623d0640 10 client.21604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:52.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc5623d0640 10 monclient: _renew_subs 2024-01-21T05:05:52.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc5623d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:52.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc5623d0640 1 -- 172.21.15.69:0/4105775790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c0709f0 con 0x7fc55c0aaed0 2024-01-21T05:05:52.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc5623d0640 1 librados: init done 2024-01-21T05:05:52.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc5623d0640 10 client.21604.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:52.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc5623d0640 15 client.21604.objecter _session_command_op_assign -1 1 2024-01-21T05:05:52.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc5623d0640 20 client.21604.objecter _get_session osd=-1 returning homeless 2024-01-21T05:05:52.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc5623d0640 10 client.21604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:52.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc5623d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:52.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.460+0000 7fc5623d0640 1 -- 172.21.15.69:0/4105775790 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc55c0a86f0 con 0x7fc55c0aaed0 2024-01-21T05:05:52.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.479+0000 7fc558ff9640 1 -- 172.21.15.69:0/4105775790 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7fc55004e400 con 0x7fc55c0aaed0 2024-01-21T05:05:52.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.480+0000 7fc558ff9640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc54806c8b0 0x7fc54806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.480+0000 7fc55bfff640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc54806c8b0 0x7fc54806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:52.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.480+0000 7fc55bfff640 10 monclient: get_auth_request con 0x7fc54806c8b0 auth_method 0 2024-01-21T05:05:52.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.481+0000 7fc558ff9640 10 client.21604.objecter ms_handle_connect 0x7fc54806c8b0 2024-01-21T05:05:52.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.481+0000 7fc55bfff640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc54806c8b0 0x7fc54806ed70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0011e0 tx=0x7fc54c007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:52.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.482+0000 7fc558ff9640 1 -- 172.21.15.69:0/4105775790 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7fc55007a470 con 0x7fc55c0aaed0 2024-01-21T05:05:52.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 10 client.21604.objecter ms_dispatch 0x7fc55c000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:52.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 3 client.21604.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:52.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 3 client.21604.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 10 client.21604.objecter checking command 1 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] conn(0x7fc5480745a0 0x7fc548076a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 20 client.21604.objecter _get_session s=0x7fc548073ed0 osd=5 3 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 20 client.21604.objecter put_session s=0x7fc548073ed0 osd=5 3 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 15 client.21604.objecter _session_command_op_remove -1 1 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 20 client.21604.objecter _get_session s=0x7fc548073ed0 osd=5 3 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 20 client.21604.objecter get_session s=0x7fc548073ed0 osd=5 3 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 15 client.21604.objecter _session_command_op_assign 5 1 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 20 client.21604.objecter put_session s=0x7fc548073ed0 osd=5 4 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 10 client.21604.objecter _send_command 1 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 1 -- 172.21.15.69:0/4105775790 --> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc548077110 con 0x7fc5480745a0 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 20 client.21604.objecter dump_active .. 0 homeless 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 1 -- 172.21.15.69:0/4105775790 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=1109) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5500767b0 con 0x7fc55c0aaed0 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc558ff9640 10 monclient: handle_get_version_reply finishing 1 version 1109 2024-01-21T05:05:52.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.483+0000 7fc56096d640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] conn(0x7fc5480745a0 0x7fc548076a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:52.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.484+0000 7fc56096d640 10 monclient: get_auth_request con 0x7fc5480745a0 auth_method 0 2024-01-21T05:05:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.487+0000 7fc56096d640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] conn(0x7fc5480745a0 0x7fc548076a40 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.487+0000 7fc558ff9640 10 client.21604.objecter ms_handle_connect 0x7fc5480745a0 2024-01-21T05:05:52.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.511+0000 7fc558ff9640 1 -- 172.21.15.69:0/4105775790 <== osd.5 v2:172.21.15.144:6802/1782989653 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc548077110 con 0x7fc5480745a0 2024-01-21T05:05:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.511+0000 7fc558ff9640 10 client.21604.objecter ms_dispatch 0x7fc55c000b90 command_reply(tid 1: 0 ) v1 2024-01-21T05:05:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.511+0000 7fc558ff9640 10 client.21604.objecter _finish_command 1 = system:0 2024-01-21T05:05:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.511+0000 7fc558ff9640 20 client.21604.objecter put_session s=0x7fc548073ed0 osd=5 4 2024-01-21T05:05:52.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.511+0000 7fc558ff9640 15 client.21604.objecter _session_command_op_remove 5 1 2024-01-21T05:05:52.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.524+0000 7fc5623d0640 10 client.21604.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T05:05:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.524+0000 7fc5623d0640 15 client.21604.objecter _session_command_op_assign -1 2 2024-01-21T05:05:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.524+0000 7fc5623d0640 20 client.21604.objecter _get_session s=0x7fc548073ed0 osd=5 3 2024-01-21T05:05:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.524+0000 7fc5623d0640 20 client.21604.objecter put_session s=0x7fc548073ed0 osd=5 3 2024-01-21T05:05:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.524+0000 7fc5623d0640 20 client.21604.objecter _get_session s=0x7fc548073ed0 osd=5 3 2024-01-21T05:05:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.524+0000 7fc5623d0640 15 client.21604.objecter _session_command_op_remove -1 2 2024-01-21T05:05:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.524+0000 7fc5623d0640 20 client.21604.objecter get_session s=0x7fc548073ed0 osd=5 3 2024-01-21T05:05:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.524+0000 7fc5623d0640 15 client.21604.objecter _session_command_op_assign 5 2 2024-01-21T05:05:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.524+0000 7fc5623d0640 20 client.21604.objecter put_session s=0x7fc548073ed0 osd=5 4 2024-01-21T05:05:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.524+0000 7fc5623d0640 10 client.21604.objecter _send_command 2 2024-01-21T05:05:52.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.524+0000 7fc5623d0640 1 -- 172.21.15.69:0/4105775790 --> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc55c066a00 con 0x7fc5480745a0 2024-01-21T05:05:52.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc558ff9640 1 -- 172.21.15.69:0/4105775790 <== osd.5 v2:172.21.15.144:6802/1782989653 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fc55400a500 con 0x7fc5480745a0 2024-01-21T05:05:52.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc558ff9640 10 client.21604.objecter ms_dispatch 0x7fc55c000b90 command_reply(tid 2: 0 ) v1 2024-01-21T05:05:52.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc558ff9640 10 client.21604.objecter _finish_command 2 = system:0 2024-01-21T05:05:52.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc558ff9640 20 client.21604.objecter put_session s=0x7fc548073ed0 osd=5 4 2024-01-21T05:05:52.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc558ff9640 15 client.21604.objecter _session_command_op_remove 5 2 2024-01-21T05:05:52.538 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-21T05:05:52.538 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T05:05:52.538 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-21T05:05:52.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 10 librados: watch_flush enter 2024-01-21T05:05:52.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 10 librados: watch_flush exit 2024-01-21T05:05:52.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 10 client.21604.objecter close_session for osd.5 2024-01-21T05:05:52.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 1 -- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] conn(0x7fc5480745a0 msgr2=0x7fc548076a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:6802/1782989653,v1:172.21.15.144:6803/1782989653] conn(0x7fc5480745a0 0x7fc548076a40 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 20 client.21604.objecter put_session s=0x7fc548073ed0 osd=5 1 2024-01-21T05:05:52.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 20 client.21604.objecter shutdown clearing up homeless session... 2024-01-21T05:05:52.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 10 client.21604.objecter successfully canceled tick 2024-01-21T05:05:52.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 1 -- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc54806c8b0 msgr2=0x7fc54806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7fc54806c8b0 0x7fc54806ed70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0011e0 tx=0x7fc54c007d30 comp rx=0 tx=0).stop 2024-01-21T05:05:52.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 10 monclient: shutdown 2024-01-21T05:05:52.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:52.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 1 -- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc55c0aaed0 msgr2=0x7fc55c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 1 --2- 172.21.15.69:0/4105775790 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc55c0aaed0 0x7fc55c06eb30 secure :-1 s=READY pgs=5235 cs=0 l=1 rev1=1 crypto rx=0x7fc5500098c0 tx=0x7fc5500046c0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc56096d640 1 -- 172.21.15.69:0/4105775790 reap_dead start 2024-01-21T05:05:52.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 1 -- 172.21.15.69:0/4105775790 shutdown_connections 2024-01-21T05:05:52.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.537+0000 7fc5623d0640 1 -- 172.21.15.69:0/4105775790 >> 172.21.15.69:0/4105775790 conn(0x7fc55c06a630 msgr2=0x7fc55c0b31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:52.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.538+0000 7fc5623d0640 1 -- 172.21.15.69:0/4105775790 shutdown_connections 2024-01-21T05:05:52.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.538+0000 7fc5623d0640 1 -- 172.21.15.69:0/4105775790 wait complete. 2024-01-21T05:05:52.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.538+0000 7fc5623d0640 1 librados: shutdown 2024-01-21T05:05:52.549 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-21T05:05:52.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.653+0000 7f39ebe56640 10 monclient: get_monmap_and_config 2024-01-21T05:05:52.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.653+0000 7f39ebe56640 10 monclient: build_initial_monmap 2024-01-21T05:05:52.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.653+0000 7f39ebe56640 10 monclient: monmap: 2024-01-21T05:05:52.654 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:52.654 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:52.654388+0000 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:52.654388+0000 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.653+0000 7f39ebe56640 1 Processor -- start 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 1 -- start start 2024-01-21T05:05:52.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 10 monclient: init 2024-01-21T05:05:52.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39e40a8b20 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f39e40a9d10 0x7f39e413f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 10 monclient(hunting): picked mon.noname-c con 0x7f39e40a9d10 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e40b2310 0x7f39e413f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 10 monclient(hunting): picked mon.noname-b con 0x7f39e40b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e41456f0 con 0x7f39e40a8b20 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e40a98c0 con 0x7f39e40a9d10 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e413fb00 con 0x7f39e40b2310 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.654+0000 7f39ebe56640 10 monclient(hunting): authenticate will time out at 3310.995958s 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.655+0000 7f39e9bf2640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f39e40a8b20 0x7f39e4145b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50258/0 (socket says 172.21.15.69:50258) 2024-01-21T05:05:52.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.655+0000 7f39e9bf2640 1 -- 172.21.15.69:0/1679985196 learned_addr learned my addr 172.21.15.69:0/1679985196 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T05:05:52.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.657+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204831597 0 0) 0x7f39e41456f0 con 0x7f39e40a8b20 2024-01-21T05:05:52.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.657+0000 7f39e8bf0640 10 monclient(hunting): _negotiate 2024-01-21T05:05:52.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.657+0000 7f39e8bf0640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:52.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.657+0000 7f39e8bf0640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:52.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.657+0000 7f39e8bf0640 10 monclient(hunting): my global_id is 17493 2024-01-21T05:05:52.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.657+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d0003520 con 0x7f39e40a8b20 2024-01-21T05:05:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 298621622 0 0) 0x7f39d0003520 con 0x7f39e40a8b20 2024-01-21T05:05:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e40b2310 msgr2=0x7f39e413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 --2- 172.21.15.69:0/1679985196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e40b2310 0x7f39e413f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 >> v2:172.21.15.144:3300/0 conn(0x7f39e40a9d10 msgr2=0x7f39e413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 --2- 172.21.15.69:0/1679985196 >> v2:172.21.15.144:3300/0 conn(0x7f39e40a9d10 0x7f39e413f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient: _finish_hunting 0 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 monclient: found mon.noname-a-legacy 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.69:6789/0 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e4146290 con 0x7f39e40a8b20 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient: _finish_auth 0 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient: _check_auth_tickets 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 125097436 0 0) 0x7f39e0001670 con 0x7f39e40a8b20 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient: dump: 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:52.662 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] but i'm connected to v1:172.21.15.69:6789/0, reconnecting 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 >> v1:172.21.15.69:6789/0 conn(0x7f39e40a8b20 legacy=0x7f39e4145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 --2- 172.21.15.69:0/1679985196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e40b2310 0x7f39d0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient(hunting): picked mon.c con 0x7f39e40b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 --2- 172.21.15.69:0/1679985196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39d00040d0 0x7f39d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient(hunting): picked mon.b con 0x7f39d00040d0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 --2- 172.21.15.69:0/1679985196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 0x7f39d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient(hunting): picked mon.a con 0x7f39d0004e20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e40a98c0 con 0x7f39d0004e20 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e413fb00 con 0x7f39d00040d0 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d00055a0 con 0x7f39e40b2310 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:52.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.660+0000 7f39e8bf0640 10 monclient(hunting): _finish_auth 0 2024-01-21T05:05:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.662+0000 7f39ea3f3640 1 --2- 172.21.15.69:0/1679985196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 0x7f39d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.665+0000 7f39ea3f3640 10 monclient(hunting): get_auth_request con 0x7f39d0004e20 auth_method 0 2024-01-21T05:05:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.665+0000 7f39ea3f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.665+0000 7f39ea3f3640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:52.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.665+0000 7f39ea3f3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-01-21T05:05:52.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.665+0000 7f39ea3f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:52.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.665+0000 7f39ea3f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:52.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.665+0000 7f39ea3f3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-01-21T05:05:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 10 monclient(hunting): handle_auth_done global_id 17493 payload 293 2024-01-21T05:05:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 1 -- 172.21.15.69:0/1679985196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e40b2310 msgr2=0x7f39d0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:52.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 1 --2- 172.21.15.69:0/1679985196 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e40b2310 0x7f39d0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 1 -- 172.21.15.69:0/1679985196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39d00040d0 msgr2=0x7f39d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 1 --2- 172.21.15.69:0/1679985196 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39d00040d0 0x7f39d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 10 monclient: _finish_hunting 0 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 1 monclient: found mon.a 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 1 -- 172.21.15.69:0/1679985196 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d0003520 con 0x7f39d0004e20 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 10 monclient: _finish_auth 0 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 10 monclient: _check_auth_tickets 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ea3f3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ebe56640 5 monclient: authenticate success, global_id 17493 2024-01-21T05:05:52.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.667+0000 7f39ebe56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:52.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.669+0000 7f39ea3f3640 1 --2- 172.21.15.69:0/1679985196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 0x7f39d000b980 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009e40 tx=0x7f39d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fde6d17084568f server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:52.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc02cd90 con 0x7f39d0004e20 2024-01-21T05:05:52.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39e8bf0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:52.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39e8bf0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:52.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39e8bf0640 10 monclient: dump: 2024-01-21T05:05:52.674 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:52.674 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:52.674 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.674 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.674 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39ebe56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39dc034440 con 0x7f39d0004e20 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39e8bf0640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39ebe56640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39e8bf0640 1 -- 172.21.15.69:0/1679985196 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39dc0347f0 con 0x7f39d0004e20 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39ebe56640 4 set_mon_vals no callback set 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39e8bf0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:52.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39e8bf0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.673+0000 7f39e8bf0640 10 monclient: dump: 2024-01-21T05:05:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:52.676 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:52.676 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.674+0000 7f39ebe56640 10 monclient: shutdown 2024-01-21T05:05:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.674+0000 7f39ebe56640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.674+0000 7f39ebe56640 1 -- 172.21.15.69:0/1679985196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 msgr2=0x7f39d000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.674+0000 7f39ebe56640 1 --2- 172.21.15.69:0/1679985196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 0x7f39d000b980 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f39dc009e40 tx=0x7f39d00057f0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.674+0000 7f39ea3f3640 1 -- 172.21.15.69:0/1679985196 reap_dead start 2024-01-21T05:05:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.674+0000 7f39ebe56640 1 -- 172.21.15.69:0/1679985196 shutdown_connections 2024-01-21T05:05:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.674+0000 7f39ebe56640 1 -- 172.21.15.69:0/1679985196 >> 172.21.15.69:0/1679985196 conn(0x7f39e4066fb0 msgr2=0x7f39e40b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.674+0000 7f39ebe56640 1 -- 172.21.15.69:0/1679985196 shutdown_connections 2024-01-21T05:05:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.674+0000 7f39ebe56640 1 -- 172.21.15.69:0/1679985196 wait complete. 2024-01-21T05:05:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.675+0000 7f39ebe56640 10 monclient: build_initial_monmap 2024-01-21T05:05:52.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.675+0000 7f39ebe56640 10 monclient: monmap: 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.675+0000 7f39ebe56640 1 librados: starting msgr at 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.675+0000 7f39ebe56640 1 librados: starting objecter 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.675+0000 7f39ebe56640 1 Processor -- start 2024-01-21T05:05:52.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.675+0000 7f39ebe56640 1 -- start start 2024-01-21T05:05:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.675+0000 7f39ebe56640 1 librados: setting wanted keys 2024-01-21T05:05:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.675+0000 7f39ebe56640 1 librados: calling monclient init 2024-01-21T05:05:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.675+0000 7f39ebe56640 10 monclient: init 2024-01-21T05:05:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 10 monclient: _add_conns ranks=[0,2,1] 2024-01-21T05:05:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 0x7f39e40adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 10 monclient(hunting): picked mon.noname-a con 0x7f39d0004e20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e40b02c0 0x7f39e40ae860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 10 monclient(hunting): picked mon.noname-c con 0x7f39e40b02c0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39e41d74c0 0x7f39e41d7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 10 monclient(hunting): picked mon.noname-b con 0x7f39e41d74c0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e41d7e90 con 0x7f39d0004e20 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e40aeda0 con 0x7f39e41d74c0 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e40afbc0 con 0x7f39e40b02c0 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39ebe56640 10 monclient(hunting): authenticate will time out at 3311.017713s 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.676+0000 7f39e9bf2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 0x7f39e40adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.677+0000 7f39e9bf2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 0x7f39e40adab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41764/0 (socket says 172.21.15.69:41764) 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.677+0000 7f39e9bf2640 1 -- 172.21.15.69:0/1659648626 learned_addr learned my addr 172.21.15.69:0/1659648626 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:52.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.677+0000 7f39e9bf2640 10 monclient(hunting): get_auth_request con 0x7f39d0004e20 auth_method 0 2024-01-21T05:05:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.677+0000 7f39e9bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.677+0000 7f39e9bf2640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.677+0000 7f39e9bf2640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.677+0000 7f39e9bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.677+0000 7f39e9bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:52.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.677+0000 7f39e9bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.680+0000 7f39e9bf2640 10 monclient(hunting): handle_auth_done global_id 17496 payload 824 2024-01-21T05:05:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.681+0000 7f39e9bf2640 1 -- 172.21.15.69:0/1659648626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e40b02c0 msgr2=0x7f39e40ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.681+0000 7f39e9bf2640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e40b02c0 0x7f39e40ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.681+0000 7f39e9bf2640 1 -- 172.21.15.69:0/1659648626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39e41d74c0 msgr2=0x7f39e41d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.681+0000 7f39e9bf2640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39e41d74c0 0x7f39e41d7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.681+0000 7f39e9bf2640 10 monclient: _finish_hunting 0 2024-01-21T05:05:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.681+0000 7f39e9bf2640 1 monclient: found mon.noname-a 2024-01-21T05:05:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.681+0000 7f39e9bf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:52.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.681+0000 7f39e9bf2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.681+0000 7f39e9bf2640 1 -- 172.21.15.69:0/1659648626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e40afea0 con 0x7f39d0004e20 2024-01-21T05:05:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.683+0000 7f39e9bf2640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 0x7f39e40adab0 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7f39e000c7d0 tx=0x7f39e000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.683+0000 7f39da7fc640 10 client.?.objecter ms_handle_connect 0x7f39d0004e20 2024-01-21T05:05:52.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.683+0000 7f39da7fc640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:52.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.686+0000 7f39da7fc640 1 -- 172.21.15.69:0/1659648626 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e00074e0 con 0x7f39d0004e20 2024-01-21T05:05:52.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.686+0000 7f39da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:52.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:52.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39da7fc640 10 monclient: dump: 2024-01-21T05:05:52.688 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:52.688 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39da7fc640 10 monclient: _finish_auth 0 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39da7fc640 10 monclient: _check_auth_tickets 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39da7fc640 1 -- 172.21.15.69:0/1659648626 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e0007640 con 0x7f39d0004e20 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 5 monclient: authenticate success, global_id 17496 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 10 monclient: _renew_subs 2024-01-21T05:05:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 1 -- 172.21.15.69:0/1659648626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e41d6fa0 con 0x7f39d0004e20 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39da7fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39da7fc640 1 -- 172.21.15.69:0/1659648626 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e0004a20 con 0x7f39d0004e20 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39da7fc640 10 monclient: dump: 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:52.690 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39dbfff640 4 set_mon_vals no callback set 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 10 client.17496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 10 monclient: _renew_subs 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 1 -- 172.21.15.69:0/1659648626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e40b0b70 con 0x7f39d0004e20 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 1 librados: init done 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:52.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.687+0000 7f39ebe56640 1 -- 172.21.15.69:0/1659648626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e404e970 con 0x7f39d0004e20 2024-01-21T05:05:52.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.709+0000 7f39da7fc640 1 -- 172.21.15.69:0/1659648626 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f39e0004c40 con 0x7f39d0004e20 2024-01-21T05:05:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.709+0000 7f39da7fc640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f39c806c8b0 0x7f39c806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.709+0000 7f39e93f1640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f39c806c8b0 0x7f39c806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.709+0000 7f39e93f1640 10 monclient: get_auth_request con 0x7f39c806c8b0 auth_method 0 2024-01-21T05:05:52.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.710+0000 7f39da7fc640 10 client.17496.objecter ms_handle_connect 0x7f39c806c8b0 2024-01-21T05:05:52.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.710+0000 7f39e93f1640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f39c806c8b0 0x7f39c806ed70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f39e4142f10 tx=0x7f39d4006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:52.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.711+0000 7f39da7fc640 1 -- 172.21.15.69:0/1659648626 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f39e00586b0 con 0x7f39d0004e20 2024-01-21T05:05:52.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.711+0000 7f39da7fc640 10 client.17496.objecter ms_dispatch 0x7f39e4000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:52.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.711+0000 7f39da7fc640 3 client.17496.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:52.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.711+0000 7f39da7fc640 3 client.17496.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:52.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.711+0000 7f39da7fc640 20 client.17496.objecter dump_active .. 0 homeless 2024-01-21T05:05:52.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.954+0000 7f39da7fc640 1 -- 172.21.15.69:0/1659648626 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f39e00542f0 con 0x7f39d0004e20 2024-01-21T05:05:52.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.954+0000 7f39da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:52.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:52.954+0000 7f39da7fc640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:53.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.156+0000 7f39ebe56640 1 -- 172.21.15.69:0/1659648626 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f39e40a9150 con 0x7f39c806c8b0 2024-01-21T05:05:53.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.157+0000 7f39da7fc640 1 -- 172.21.15.69:0/1659648626 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f39d4013430 con 0x7f39c806c8b0 2024-01-21T05:05:53.158 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 5 to deep-scrub 2024-01-21T05:05:53.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.158+0000 7f39ebe56640 10 librados: watch_flush enter 2024-01-21T05:05:53.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.158+0000 7f39ebe56640 10 librados: watch_flush exit 2024-01-21T05:05:53.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.158+0000 7f39ebe56640 20 client.17496.objecter shutdown clearing up homeless session... 2024-01-21T05:05:53.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.158+0000 7f39ebe56640 10 client.17496.objecter successfully canceled tick 2024-01-21T05:05:53.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 -- 172.21.15.69:0/1659648626 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f39c806c8b0 msgr2=0x7f39c806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f39c806c8b0 0x7f39c806ed70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f39e4142f10 tx=0x7f39d4006c90 comp rx=0 tx=0).stop 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 10 monclient: shutdown 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 -- 172.21.15.69:0/1659648626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 msgr2=0x7f39e40adab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 0x7f39e40adab0 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7f39e000c7d0 tx=0x7f39e000cc90 comp rx=0 tx=0).stop 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 -- 172.21.15.69:0/1659648626 shutdown_connections 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f39c806c8b0 0x7f39c806ed70 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f39e41d74c0 0x7f39e41d7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f39e40b02c0 0x7f39e40ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 --2- 172.21.15.69:0/1659648626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39d0004e20 0x7f39e40adab0 unknown :-1 s=CLOSED pgs=3502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 -- 172.21.15.69:0/1659648626 >> 172.21.15.69:0/1659648626 conn(0x7f39e4066fb0 msgr2=0x7f39e413cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 -- 172.21.15.69:0/1659648626 shutdown_connections 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 -- 172.21.15.69:0/1659648626 wait complete. 2024-01-21T05:05:53.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.159+0000 7f39ebe56640 1 librados: shutdown 2024-01-21T05:05:53.171 INFO:tasks.ceph:Scrubbing osd.6 2024-01-21T05:05:53.171 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-21T05:05:53.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.274+0000 7f220bba4640 10 monclient: get_monmap_and_config 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.274+0000 7f220bba4640 10 monclient: build_initial_monmap 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.274+0000 7f220bba4640 10 monclient: monmap: 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:53.275210+0000 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:53.275210+0000 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:53.275 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.274+0000 7f220bba4640 1 Processor -- start 2024-01-21T05:05:53.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 1 -- start start 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 10 monclient: init 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2204066620 0x7f22040afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 10 monclient(hunting): picked mon.noname-a con 0x7f2204066620 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2204072720 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2204071c80 0x7f22040b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 10 monclient(hunting): picked mon.noname-b con 0x7f2204071c80 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22040ab7c0 con 0x7f2204072720 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22040ad210 con 0x7f2204066620 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2204066a00 con 0x7f2204071c80 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:53.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.275+0000 7f220bba4640 10 monclient(hunting): authenticate will time out at 3311.616849s 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f2209940640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2204066620 0x7f22040afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f220a141640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2204071c80 0x7f22040b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f220913f640 1 --1- >> v1:172.21.15.144:6789/0 conn(0x7f2204072720 0x7f220413d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.144:6789/0 says I am v1:172.21.15.69:39440/0 (socket says 172.21.15.69:39440) 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f220913f640 1 -- 172.21.15.69:0/1614461722 learned_addr learned my addr 172.21.15.69:0/1614461722 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f2209940640 10 monclient(hunting): get_auth_request con 0x7f2204066620 auth_method 0 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f2209940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f2209940640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f2209940640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f220a141640 10 monclient(hunting): get_auth_request con 0x7f2204071c80 auth_method 0 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f220a141640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f220a141640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.276+0000 7f220a141640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.277+0000 7f2209940640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:53.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.277+0000 7f2209940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:53.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.277+0000 7f2209940640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:53.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.277+0000 7f220a141640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:53.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.277+0000 7f220a141640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:53.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.277+0000 7f220a141640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:53.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.278+0000 7f2209940640 10 monclient(hunting): handle_auth_done global_id 17499 payload 274 2024-01-21T05:05:53.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.278+0000 7f220893e640 1 -- 172.21.15.69:0/1614461722 <== mon.1 v1:172.21.15.144:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179414877 0 0) 0x7f22040ab7c0 con 0x7f2204072720 2024-01-21T05:05:53.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.278+0000 7f2209940640 1 -- 172.21.15.69:0/1614461722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2204071c80 msgr2=0x7f22040b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:53.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.278+0000 7f2209940640 1 --2- 172.21.15.69:0/1614461722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2204071c80 0x7f22040b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.278+0000 7f2209940640 1 -- 172.21.15.69:0/1614461722 >> v1:172.21.15.144:6789/0 conn(0x7f2204072720 legacy=0x7f220413d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.278+0000 7f2209940640 10 monclient: _finish_hunting 0 2024-01-21T05:05:53.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.278+0000 7f2209940640 1 monclient: found mon.noname-a 2024-01-21T05:05:53.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.278+0000 7f2209940640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:53.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.278+0000 7f2209940640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:53.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.278+0000 7f2209940640 1 -- 172.21.15.69:0/1614461722 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22040ab450 con 0x7f2204066620 2024-01-21T05:05:53.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.278+0000 7f220893e640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-01-21T05:05:53.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.280+0000 7f2209940640 1 --2- 172.21.15.69:0/1614461722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2204066620 0x7f22040afbe0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009d90 tx=0x7f21fc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b3c2a86e9543ba server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:53.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 1 -- 172.21.15.69:0/1614461722 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc02cbc0 con 0x7f2204066620 2024-01-21T05:05:53.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:53.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:53.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 10 monclient: dump: 2024-01-21T05:05:53.285 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:53.285 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:53.285 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.285 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.285 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:53.285 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.285 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 10 monclient: _finish_auth 0 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 10 monclient: _check_auth_tickets 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 1 -- 172.21.15.69:0/1614461722 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc02cd20 con 0x7f2204066620 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 1 -- 172.21.15.69:0/1614461722 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc034660 con 0x7f2204066620 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220893e640 10 monclient: dump: 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:53.286 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:53.287 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.287 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.287 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:53.287 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.287 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:53.287 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:53.287 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:53.287 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220bba4640 5 monclient: authenticate success, global_id 17499 2024-01-21T05:05:53.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220bba4640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:53.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220bba4640 4 set_mon_vals no callback set 2024-01-21T05:05:53.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220bba4640 10 monclient: shutdown 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220bba4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220bba4640 1 -- 172.21.15.69:0/1614461722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2204066620 msgr2=0x7f22040afbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.284+0000 7f220bba4640 1 --2- 172.21.15.69:0/1614461722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2204066620 0x7f22040afbe0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009d90 tx=0x7f21fc02be10 comp rx=0 tx=0).stop 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.285+0000 7f220bba4640 1 -- 172.21.15.69:0/1614461722 shutdown_connections 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.285+0000 7f220bba4640 1 --2- 172.21.15.69:0/1614461722 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2204071c80 0x7f22040b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.285+0000 7f220bba4640 1 --2- 172.21.15.69:0/1614461722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2204066620 0x7f22040afbe0 unknown :-1 s=CLOSED pgs=3503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.285+0000 7f220bba4640 1 -- 172.21.15.69:0/1614461722 >> 172.21.15.69:0/1614461722 conn(0x7f220406a630 msgr2=0x7f220406aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.285+0000 7f220bba4640 1 -- 172.21.15.69:0/1614461722 shutdown_connections 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.285+0000 7f220bba4640 1 -- 172.21.15.69:0/1614461722 wait complete. 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.285+0000 7f220bba4640 10 monclient: build_initial_monmap 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.285+0000 7f220bba4640 10 monclient: monmap: 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:53.291 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 librados: starting msgr at 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 librados: starting objecter 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 Processor -- start 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 -- start start 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 librados: setting wanted keys 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 librados: calling monclient init 2024-01-21T05:05:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 10 monclient: init 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2204066620 0x7f220413f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 10 monclient(hunting): picked mon.noname-c con 0x7f2204066620 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2204071c80 0x7f2204137030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 10 monclient(hunting): picked mon.noname-a con 0x7f2204071c80 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2204137570 0x7f22040a8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 10 monclient(hunting): picked mon.noname-b con 0x7f2204137570 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2204140950 con 0x7f2204071c80 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2204140c30 con 0x7f2204137570 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220413fff0 con 0x7f2204066620 2024-01-21T05:05:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.286+0000 7f220bba4640 10 monclient(hunting): authenticate will time out at 3311.627941s 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220913f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2204071c80 0x7f2204137030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f2209940640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2204066620 0x7f220413f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220a141640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2204137570 0x7f22040a8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220913f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2204071c80 0x7f2204137030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41794/0 (socket says 172.21.15.69:41794) 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220913f640 1 -- 172.21.15.69:0/192685624 learned_addr learned my addr 172.21.15.69:0/192685624 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220913f640 10 monclient(hunting): get_auth_request con 0x7f2204071c80 auth_method 0 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220913f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220913f640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220913f640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f2209940640 10 monclient(hunting): get_auth_request con 0x7f2204066620 auth_method 0 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f2209940640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f2209940640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f2209940640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220a141640 10 monclient(hunting): get_auth_request con 0x7f2204137570 auth_method 0 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220a141640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220a141640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.287+0000 7f220a141640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.288+0000 7f220913f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.288+0000 7f220913f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.288+0000 7f220913f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.288+0000 7f2209940640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.288+0000 7f2209940640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.288+0000 7f2209940640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.288+0000 7f220a141640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.288+0000 7f220a141640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.288+0000 7f220a141640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:53.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.290+0000 7f220913f640 10 monclient(hunting): handle_auth_done global_id 17502 payload 824 2024-01-21T05:05:53.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.290+0000 7f220913f640 1 -- 172.21.15.69:0/192685624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2204066620 msgr2=0x7f220413f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.290+0000 7f220913f640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2204066620 0x7f220413f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.290+0000 7f220913f640 1 -- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2204137570 msgr2=0x7f22040a8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.290+0000 7f220913f640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2204137570 0x7f22040a8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.290+0000 7f220913f640 10 monclient: _finish_hunting 0 2024-01-21T05:05:53.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.290+0000 7f220913f640 1 monclient: found mon.noname-a 2024-01-21T05:05:53.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.290+0000 7f220913f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:53.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.290+0000 7f220913f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:53.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.290+0000 7f220913f640 1 -- 172.21.15.69:0/192685624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc009a70 con 0x7f2204071c80 2024-01-21T05:05:53.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.293+0000 7f220913f640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2204071c80 0x7f2204137030 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7f21f800c430 tx=0x7f21f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:53.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.293+0000 7f22027fc640 10 client.?.objecter ms_handle_connect 0x7f2204071c80 2024-01-21T05:05:53.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.293+0000 7f22027fc640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:53.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 1 -- 172.21.15.69:0/192685624 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f80074e0 con 0x7f2204071c80 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 10 monclient: dump: 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:53.299 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 10 monclient: _finish_auth 0 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 10 monclient: _check_auth_tickets 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 1 -- 172.21.15.69:0/192685624 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f8007660 con 0x7f2204071c80 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 1 -- 172.21.15.69:0/192685624 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8004a20 con 0x7f2204071c80 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f220bba4640 5 monclient: authenticate success, global_id 17502 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f220bba4640 10 monclient: _renew_subs 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f220bba4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f220bba4640 1 -- 172.21.15.69:0/192685624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22041382d0 con 0x7f2204071c80 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f22027fc640 10 monclient: dump: 2024-01-21T05:05:53.300 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.296+0000 7f2203fff640 4 set_mon_vals no callback set 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.297+0000 7f220bba4640 10 client.17502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.297+0000 7f220bba4640 10 monclient: _renew_subs 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.297+0000 7f220bba4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:53.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.297+0000 7f220bba4640 1 -- 172.21.15.69:0/192685624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2204137a00 con 0x7f2204071c80 2024-01-21T05:05:53.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.297+0000 7f220bba4640 1 librados: init done 2024-01-21T05:05:53.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.297+0000 7f220bba4640 10 client.17502.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:53.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.297+0000 7f220bba4640 15 client.17502.objecter _session_command_op_assign -1 1 2024-01-21T05:05:53.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.297+0000 7f220bba4640 20 client.17502.objecter _get_session osd=-1 returning homeless 2024-01-21T05:05:53.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.297+0000 7f220bba4640 10 client.17502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:53.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.297+0000 7f220bba4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:53.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.297+0000 7f220bba4640 1 -- 172.21.15.69:0/192685624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f220413fe30 con 0x7f2204071c80 2024-01-21T05:05:53.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.316+0000 7f22027fc640 1 -- 172.21.15.69:0/192685624 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f21f80133c0 con 0x7f2204071c80 2024-01-21T05:05:53.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.316+0000 7f22027fc640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f21f006c8b0 0x7f21f006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.316+0000 7f2209940640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f21f006c8b0 0x7f21f006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.316+0000 7f2209940640 10 monclient: get_auth_request con 0x7f21f006c8b0 auth_method 0 2024-01-21T05:05:53.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.317+0000 7f2209940640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f21f006c8b0 0x7f21f006ed70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f21fc036000 tx=0x7f21fc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:53.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.317+0000 7f22027fc640 10 client.17502.objecter ms_handle_connect 0x7f21f006c8b0 2024-01-21T05:05:53.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 1 -- 172.21.15.69:0/192685624 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f21f8058870 con 0x7f2204071c80 2024-01-21T05:05:53.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 10 client.17502.objecter ms_dispatch 0x7f2204000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:53.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 3 client.17502.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:53.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 3 client.17502.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:53.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 10 client.17502.objecter checking command 1 2024-01-21T05:05:53.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] conn(0x7f21f00745a0 0x7f21f0076a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 20 client.17502.objecter _get_session s=0x7f21f0073ed0 osd=6 3 2024-01-21T05:05:53.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 20 client.17502.objecter put_session s=0x7f21f0073ed0 osd=6 3 2024-01-21T05:05:53.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 15 client.17502.objecter _session_command_op_remove -1 1 2024-01-21T05:05:53.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 20 client.17502.objecter _get_session s=0x7f21f0073ed0 osd=6 3 2024-01-21T05:05:53.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 20 client.17502.objecter get_session s=0x7f21f0073ed0 osd=6 3 2024-01-21T05:05:53.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 15 client.17502.objecter _session_command_op_assign 6 1 2024-01-21T05:05:53.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 20 client.17502.objecter put_session s=0x7f21f0073ed0 osd=6 4 2024-01-21T05:05:53.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 10 client.17502.objecter _send_command 1 2024-01-21T05:05:53.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 1 -- 172.21.15.69:0/192685624 --> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21f0077110 con 0x7f21f00745a0 2024-01-21T05:05:53.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.318+0000 7f22027fc640 20 client.17502.objecter dump_active .. 0 homeless 2024-01-21T05:05:53.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.319+0000 7f220a141640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] conn(0x7f21f00745a0 0x7f21f0076a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.319+0000 7f220a141640 10 monclient: get_auth_request con 0x7f21f00745a0 auth_method 0 2024-01-21T05:05:53.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.320+0000 7f22027fc640 1 -- 172.21.15.69:0/192685624 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=1109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21f8001170 con 0x7f2204071c80 2024-01-21T05:05:53.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.320+0000 7f22027fc640 10 monclient: handle_get_version_reply finishing 1 version 1109 2024-01-21T05:05:53.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.322+0000 7f22027fc640 10 client.17502.objecter ms_handle_connect 0x7f21f00745a0 2024-01-21T05:05:53.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.322+0000 7f220a141640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] conn(0x7f21f00745a0 0x7f21f0076a40 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:53.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.346+0000 7f22027fc640 1 -- 172.21.15.69:0/192685624 <== osd.6 v2:172.21.15.144:6812/479183693 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21f0077110 con 0x7f21f00745a0 2024-01-21T05:05:53.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.346+0000 7f22027fc640 10 client.17502.objecter ms_dispatch 0x7f2204000b90 command_reply(tid 1: 0 ) v1 2024-01-21T05:05:53.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.346+0000 7f22027fc640 10 client.17502.objecter _finish_command 1 = system:0 2024-01-21T05:05:53.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.346+0000 7f22027fc640 20 client.17502.objecter put_session s=0x7f21f0073ed0 osd=6 4 2024-01-21T05:05:53.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.346+0000 7f22027fc640 15 client.17502.objecter _session_command_op_remove 6 1 2024-01-21T05:05:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.359+0000 7f220bba4640 10 client.17502.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T05:05:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.359+0000 7f220bba4640 15 client.17502.objecter _session_command_op_assign -1 2 2024-01-21T05:05:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.359+0000 7f220bba4640 20 client.17502.objecter _get_session s=0x7f21f0073ed0 osd=6 3 2024-01-21T05:05:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.359+0000 7f220bba4640 20 client.17502.objecter put_session s=0x7f21f0073ed0 osd=6 3 2024-01-21T05:05:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.359+0000 7f220bba4640 20 client.17502.objecter _get_session s=0x7f21f0073ed0 osd=6 3 2024-01-21T05:05:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.359+0000 7f220bba4640 15 client.17502.objecter _session_command_op_remove -1 2 2024-01-21T05:05:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.359+0000 7f220bba4640 20 client.17502.objecter get_session s=0x7f21f0073ed0 osd=6 3 2024-01-21T05:05:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.359+0000 7f220bba4640 15 client.17502.objecter _session_command_op_assign 6 2 2024-01-21T05:05:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.359+0000 7f220bba4640 20 client.17502.objecter put_session s=0x7f21f0073ed0 osd=6 4 2024-01-21T05:05:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.359+0000 7f220bba4640 10 client.17502.objecter _send_command 2 2024-01-21T05:05:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.359+0000 7f220bba4640 1 -- 172.21.15.69:0/192685624 --> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2204072970 con 0x7f21f00745a0 2024-01-21T05:05:53.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.371+0000 7f22027fc640 1 -- 172.21.15.69:0/192685624 <== osd.6 v2:172.21.15.144:6812/479183693 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2204072970 con 0x7f21f00745a0 2024-01-21T05:05:53.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.371+0000 7f22027fc640 10 client.17502.objecter ms_dispatch 0x7f2204000b90 command_reply(tid 2: 0 ) v1 2024-01-21T05:05:53.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.371+0000 7f22027fc640 10 client.17502.objecter _finish_command 2 = system:0 2024-01-21T05:05:53.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.371+0000 7f22027fc640 20 client.17502.objecter put_session s=0x7f21f0073ed0 osd=6 4 2024-01-21T05:05:53.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.371+0000 7f22027fc640 15 client.17502.objecter _session_command_op_remove 6 2 2024-01-21T05:05:53.373 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-21T05:05:53.373 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T05:05:53.373 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-21T05:05:53.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 10 librados: watch_flush enter 2024-01-21T05:05:53.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 10 librados: watch_flush exit 2024-01-21T05:05:53.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 10 client.17502.objecter close_session for osd.6 2024-01-21T05:05:53.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 1 -- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] conn(0x7f21f00745a0 msgr2=0x7f21f0076a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:6812/479183693,v1:172.21.15.144:6813/479183693] conn(0x7f21f00745a0 0x7f21f0076a40 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 20 client.17502.objecter put_session s=0x7f21f0073ed0 osd=6 1 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 20 client.17502.objecter shutdown clearing up homeless session... 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 10 client.17502.objecter successfully canceled tick 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 1 -- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f21f006c8b0 msgr2=0x7f21f006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f21f006c8b0 0x7f21f006ed70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f21fc036000 tx=0x7f21fc030000 comp rx=0 tx=0).stop 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 10 monclient: shutdown 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 1 -- 172.21.15.69:0/192685624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2204071c80 msgr2=0x7f2204137030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 1 --2- 172.21.15.69:0/192685624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2204071c80 0x7f2204137030 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7f21f800c430 tx=0x7f21f800c8f0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220a141640 1 -- 172.21.15.69:0/192685624 reap_dead start 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 1 -- 172.21.15.69:0/192685624 shutdown_connections 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 1 -- 172.21.15.69:0/192685624 >> 172.21.15.69:0/192685624 conn(0x7f220406a630 msgr2=0x7f220413c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 1 -- 172.21.15.69:0/192685624 shutdown_connections 2024-01-21T05:05:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 1 -- 172.21.15.69:0/192685624 wait complete. 2024-01-21T05:05:53.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.372+0000 7f220bba4640 1 librados: shutdown 2024-01-21T05:05:53.383 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-21T05:05:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.482+0000 7f85030ec640 10 monclient: get_monmap_and_config 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.482+0000 7f85030ec640 10 monclient: build_initial_monmap 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.482+0000 7f85030ec640 10 monclient: monmap: 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:53.483080+0000 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:53.483080+0000 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:53.483 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.482+0000 7f85030ec640 1 Processor -- start 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.482+0000 7f85030ec640 1 -- start start 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.482+0000 7f85030ec640 10 monclient: init 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84fc0a8b20 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0a9d10 0x7f84fc13f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 10 monclient(hunting): picked mon.noname-b con 0x7f84fc0a9d10 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f84fc0b2310 0x7f84fc13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 10 monclient(hunting): picked mon.noname-a con 0x7f84fc0b2310 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84fc1456f0 con 0x7f84fc0a8b20 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84fc0a98c0 con 0x7f84fc0b2310 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc13fb00 con 0x7f84fc0a9d10 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.483+0000 7f85030ec640 10 monclient(hunting): authenticate will time out at 3311.824751s 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f8501689640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f84fc0b2310 0x7f84fc13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0a9d10 0x7f84fc13f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f8501689640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f84fc0b2310 0x7f84fc13f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41802/0 (socket says 172.21.15.69:41802) 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f8501689640 1 -- 172.21.15.69:0/3610034565 learned_addr learned my addr 172.21.15.69:0/3610034565 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f8501689640 10 monclient(hunting): get_auth_request con 0x7f84fc0b2310 auth_method 0 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f8501689640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f8501689640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f8501689640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f84fbfff640 10 monclient(hunting): get_auth_request con 0x7f84fc0a9d10 auth_method 0 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f84fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f84fbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.484+0000 7f84fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:53.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.485+0000 7f84fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.485+0000 7f84fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.485+0000 7f84fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.485+0000 7f8501689640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.485+0000 7f8501689640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.485+0000 7f8501689640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.486+0000 7f84fbfff640 10 monclient(hunting): handle_auth_done global_id 18875 payload 274 2024-01-21T05:05:53.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.486+0000 7f84fbfff640 1 -- 172.21.15.69:0/3610034565 >> v2:172.21.15.69:3300/0 conn(0x7f84fc0b2310 msgr2=0x7f84fc13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:53.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.486+0000 7f84fbfff640 1 --2- 172.21.15.69:0/3610034565 >> v2:172.21.15.69:3300/0 conn(0x7f84fc0b2310 0x7f84fc13f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.486+0000 7f84fbfff640 1 -- 172.21.15.69:0/3610034565 >> v1:172.21.15.69:6789/0 conn(0x7f84fc0a8b20 legacy=0x7f84fc145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:53.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.486+0000 7f84fbfff640 10 monclient: _finish_hunting 0 2024-01-21T05:05:53.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.486+0000 7f84fbfff640 1 monclient: found mon.noname-b 2024-01-21T05:05:53.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.486+0000 7f84fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:53.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.486+0000 7f84fbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.69:3301/0 2024-01-21T05:05:53.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.486+0000 7f84fbfff640 1 -- 172.21.15.69:0/3610034565 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc146290 con 0x7f84fc0a9d10 2024-01-21T05:05:53.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.488+0000 7f84fbfff640 1 --2- 172.21.15.69:0/3610034565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0a9d10 0x7f84fc13f080 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7f84f00097a0 tx=0x7f84f002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a413fc3327d7b53c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 1 -- 172.21.15.69:0/3610034565 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f002e070 con 0x7f84fc0a9d10 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 10 monclient: dump: 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:53.494 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 10 monclient: _finish_auth 0 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 10 monclient: _check_auth_tickets 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 1 -- 172.21.15.69:0/3610034565 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f00315a0 con 0x7f84fc0a9d10 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f85030ec640 5 monclient: authenticate success, global_id 18875 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f85030ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f85030ec640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 1 -- 172.21.15.69:0/3610034565 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0031930 con 0x7f84fc0a9d10 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f85030ec640 4 set_mon_vals no callback set 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T05:05:53.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f84fb7fe640 10 monclient: dump: 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f85030ec640 10 monclient: shutdown 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f85030ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f85030ec640 1 -- 172.21.15.69:0/3610034565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0a9d10 msgr2=0x7f84fc13f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.492+0000 7f85030ec640 1 --2- 172.21.15.69:0/3610034565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0a9d10 0x7f84fc13f080 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7f84f00097a0 tx=0x7f84f002b840 comp rx=0 tx=0).stop 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.493+0000 7f85030ec640 1 -- 172.21.15.69:0/3610034565 shutdown_connections 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.493+0000 7f85030ec640 1 --2- 172.21.15.69:0/3610034565 >> v2:172.21.15.69:3300/0 conn(0x7f84fc0b2310 0x7f84fc13f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.493+0000 7f85030ec640 1 --2- 172.21.15.69:0/3610034565 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0a9d10 0x7f84fc13f080 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.493+0000 7f85030ec640 1 -- 172.21.15.69:0/3610034565 >> 172.21.15.69:0/3610034565 conn(0x7f84fc066fb0 msgr2=0x7f84fc0b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.493+0000 7f85030ec640 1 -- 172.21.15.69:0/3610034565 shutdown_connections 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.493+0000 7f85030ec640 1 -- 172.21.15.69:0/3610034565 wait complete. 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.493+0000 7f85030ec640 10 monclient: build_initial_monmap 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.493+0000 7f85030ec640 10 monclient: monmap: 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:53.498 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.493+0000 7f85030ec640 1 librados: starting msgr at 2024-01-21T05:05:53.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.493+0000 7f85030ec640 1 librados: starting objecter 2024-01-21T05:05:53.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 1 Processor -- start 2024-01-21T05:05:53.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 1 -- start start 2024-01-21T05:05:53.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 1 librados: setting wanted keys 2024-01-21T05:05:53.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 1 librados: calling monclient init 2024-01-21T05:05:53.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 10 monclient: init 2024-01-21T05:05:53.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:53.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 10 monclient: _add_conns ranks=[1,0,2] 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84fc0a8b20 0x7f84fc1409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 10 monclient(hunting): picked mon.noname-b con 0x7f84fc0a8b20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84fc0a9d10 0x7f84fc140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 10 monclient(hunting): picked mon.noname-a con 0x7f84fc0a9d10 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0b2310 0x7f84fc13a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 10 monclient(hunting): picked mon.noname-c con 0x7f84fc0b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc1416c0 con 0x7f84fc0a9d10 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc142b50 con 0x7f84fc0a8b20 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc13fd60 con 0x7f84fc0b2310 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.494+0000 7f85030ec640 10 monclient(hunting): authenticate will time out at 3311.836108s 2024-01-21T05:05:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8501689640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0b2310 0x7f84fc13a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84fc0a9d10 0x7f84fc140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8500e88640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84fc0a8b20 0x7f84fc1409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8501689640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0b2310 0x7f84fc13a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:48736/0 (socket says 172.21.15.69:48736) 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8501689640 1 -- 172.21.15.69:0/1191354692 learned_addr learned my addr 172.21.15.69:0/1191354692 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8501689640 10 monclient(hunting): get_auth_request con 0x7f84fc0b2310 auth_method 0 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8501689640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8501689640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8501689640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f84fbfff640 10 monclient(hunting): get_auth_request con 0x7f84fc0a9d10 auth_method 0 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f84fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f84fbfff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f84fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8500e88640 10 monclient(hunting): get_auth_request con 0x7f84fc0a8b20 auth_method 0 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8500e88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8500e88640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:53.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.495+0000 7f8500e88640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.496+0000 7f8501689640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.496+0000 7f8501689640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.496+0000 7f8501689640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.496+0000 7f84fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.496+0000 7f84fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.496+0000 7f84fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.496+0000 7f8500e88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.496+0000 7f8500e88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.496+0000 7f8500e88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.499+0000 7f8501689640 10 monclient(hunting): handle_auth_done global_id 18878 payload 824 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.499+0000 7f8501689640 1 -- 172.21.15.69:0/1191354692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84fc0a8b20 msgr2=0x7f84fc1409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.499+0000 7f8501689640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84fc0a8b20 0x7f84fc1409d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.499+0000 7f8501689640 1 -- 172.21.15.69:0/1191354692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84fc0a9d10 msgr2=0x7f84fc140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.499+0000 7f8501689640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84fc0a9d10 0x7f84fc140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.499+0000 7f8501689640 10 monclient: _finish_hunting 0 2024-01-21T05:05:53.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.499+0000 7f8501689640 1 monclient: found mon.noname-c 2024-01-21T05:05:53.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.499+0000 7f8501689640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:53.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.499+0000 7f8501689640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.69:3301/0 2024-01-21T05:05:53.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.499+0000 7f8501689640 1 -- 172.21.15.69:0/1191354692 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f0009400 con 0x7f84fc0b2310 2024-01-21T05:05:53.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.499+0000 7f84fbfff640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84fc0a9d10 0x7f84fc140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-21T05:05:53.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.501+0000 7f8501689640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0b2310 0x7f84fc13a090 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7f84f400efc0 tx=0x7f84f400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:53.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.501+0000 7f84f97fa640 10 client.?.objecter ms_handle_connect 0x7f84fc0b2310 2024-01-21T05:05:53.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.501+0000 7f84f97fa640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:53.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 1 -- 172.21.15.69:0/1191354692 <== mon.2 v2:172.21.15.69:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4009bd0 con 0x7f84fc0b2310 2024-01-21T05:05:53.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:53.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 10 monclient: dump: 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 10 monclient: _finish_auth 0 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 10 monclient: _check_auth_tickets 2024-01-21T05:05:53.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 1 -- 172.21.15.69:0/1191354692 <== mon.2 v2:172.21.15.69:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f400f040 con 0x7f84fc0b2310 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 1 -- 172.21.15.69:0/1191354692 <== mon.2 v2:172.21.15.69:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4003990 con 0x7f84fc0b2310 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84f97fa640 10 monclient: dump: 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:53.508 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f85030ec640 5 monclient: authenticate success, global_id 18878 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f85030ec640 10 monclient: _renew_subs 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f85030ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f85030ec640 1 -- 172.21.15.69:0/1191354692 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84fc141b50 con 0x7f84fc0b2310 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f84faffd640 4 set_mon_vals no callback set 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f85030ec640 10 client.18878.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f85030ec640 10 monclient: _renew_subs 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f85030ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f85030ec640 1 -- 172.21.15.69:0/1191354692 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84fc13a7b0 con 0x7f84fc0b2310 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.504+0000 7f85030ec640 1 librados: init done 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.505+0000 7f85030ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:53.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.505+0000 7f85030ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:53.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.505+0000 7f85030ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.69:3301/0 2024-01-21T05:05:53.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.505+0000 7f85030ec640 1 -- 172.21.15.69:0/1191354692 --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84fc13aa00 con 0x7f84fc0b2310 2024-01-21T05:05:53.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.524+0000 7f84f97fa640 1 -- 172.21.15.69:0/1191354692 <== mon.2 v2:172.21.15.69:3301/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f84f4003af0 con 0x7f84fc0b2310 2024-01-21T05:05:53.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.525+0000 7f84f97fa640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f84e806c8b0 0x7f84e806ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:53.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.525+0000 7f8500e88640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f84e806c8b0 0x7f84e806ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:53.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.525+0000 7f8500e88640 10 monclient: get_auth_request con 0x7f84e806c8b0 auth_method 0 2024-01-21T05:05:53.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.525+0000 7f84f97fa640 10 client.18878.objecter ms_handle_connect 0x7f84e806c8b0 2024-01-21T05:05:53.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.525+0000 7f8500e88640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f84e806c8b0 0x7f84e806ed70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006fd0 tx=0x7f84ec009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:53.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.526+0000 7f84f97fa640 1 -- 172.21.15.69:0/1191354692 <== mon.2 v2:172.21.15.69:3301/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f84f4059530 con 0x7f84fc0b2310 2024-01-21T05:05:53.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.526+0000 7f84f97fa640 10 client.18878.objecter ms_dispatch 0x7f84fc000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:53.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.526+0000 7f84f97fa640 3 client.18878.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:53.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.526+0000 7f84f97fa640 3 client.18878.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:53.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.526+0000 7f84f97fa640 20 client.18878.objecter dump_active .. 0 homeless 2024-01-21T05:05:53.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.753+0000 7f84f97fa640 1 -- 172.21.15.69:0/1191354692 <== mon.2 v2:172.21.15.69:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f84f4055870 con 0x7f84fc0b2310 2024-01-21T05:05:53.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.753+0000 7f84f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:53.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.753+0000 7f84f97fa640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:53.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.929+0000 7f85030ec640 1 -- 172.21.15.69:0/1191354692 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f84fc142cc0 con 0x7f84e806c8b0 2024-01-21T05:05:53.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.929+0000 7f84f97fa640 1 -- 172.21.15.69:0/1191354692 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f84fc142cc0 con 0x7f84e806c8b0 2024-01-21T05:05:53.930 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 6 to deep-scrub 2024-01-21T05:05:53.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 10 librados: watch_flush enter 2024-01-21T05:05:53.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 10 librados: watch_flush exit 2024-01-21T05:05:53.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 20 client.18878.objecter shutdown clearing up homeless session... 2024-01-21T05:05:53.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 10 client.18878.objecter successfully canceled tick 2024-01-21T05:05:53.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 1 -- 172.21.15.69:0/1191354692 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f84e806c8b0 msgr2=0x7f84e806ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f84e806c8b0 0x7f84e806ed70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006fd0 tx=0x7f84ec009060 comp rx=0 tx=0).stop 2024-01-21T05:05:53.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 10 monclient: shutdown 2024-01-21T05:05:53.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:53.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 1 -- 172.21.15.69:0/1191354692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0b2310 msgr2=0x7f84fc13a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:53.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0b2310 0x7f84fc13a090 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7f84f400efc0 tx=0x7f84f400c490 comp rx=0 tx=0).stop 2024-01-21T05:05:53.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 1 -- 172.21.15.69:0/1191354692 shutdown_connections 2024-01-21T05:05:53.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f84e806c8b0 0x7f84e806ed70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f84fc0b2310 0x7f84fc13a090 unknown :-1 s=CLOSED pgs=3328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84fc0a9d10 0x7f84fc140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 1 --2- 172.21.15.69:0/1191354692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f84fc0a8b20 0x7f84fc1409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:53.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.931+0000 7f85030ec640 1 -- 172.21.15.69:0/1191354692 >> 172.21.15.69:0/1191354692 conn(0x7f84fc066fb0 msgr2=0x7f84fc13ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:53.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.932+0000 7f85030ec640 1 -- 172.21.15.69:0/1191354692 shutdown_connections 2024-01-21T05:05:53.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.932+0000 7f85030ec640 1 -- 172.21.15.69:0/1191354692 wait complete. 2024-01-21T05:05:53.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:53.932+0000 7f85030ec640 1 librados: shutdown 2024-01-21T05:05:53.943 INFO:tasks.ceph:Scrubbing osd.7 2024-01-21T05:05:53.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-21T05:05:54.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.080+0000 7f079a93e640 10 monclient: get_monmap_and_config 2024-01-21T05:05:54.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.080+0000 7f079a93e640 10 monclient: build_initial_monmap 2024-01-21T05:05:54.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.080+0000 7f079a93e640 10 monclient: monmap: 2024-01-21T05:05:54.081 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:54.081 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:54.082 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:54.081902+0000 2024-01-21T05:05:54.082 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:54.081902+0000 2024-01-21T05:05:54.082 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:54.082 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.082 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:54.082 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:54.082 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:54.082 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:54.082 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:54.082 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.081+0000 7f079a93e640 1 Processor -- start 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.081+0000 7f079a93e640 1 -- start start 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.081+0000 7f079a93e640 10 monclient: init 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07940a8b20 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f07940a9d10 0x7f079413f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 10 monclient(hunting): picked mon.noname-a con 0x7f07940a9d10 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07940b2310 0x7f079413f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 10 monclient(hunting): picked mon.noname-c con 0x7f07940b2310 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07941456f0 con 0x7f07940a8b20 2024-01-21T05:05:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07940a98c0 con 0x7f07940a9d10 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079413fb00 con 0x7f07940b2310 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.082+0000 7f079a93e640 10 monclient(hunting): authenticate will time out at 3312.423515s 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.083+0000 7f0798edb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07940b2310 0x7f079413f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.083+0000 7f0798edb640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f07940b2310 0x7f079413f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:41458/0 (socket says 172.21.15.69:41458) 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.083+0000 7f0798edb640 1 -- 172.21.15.69:0/625855175 learned_addr learned my addr 172.21.15.69:0/625855175 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.083+0000 7f0798edb640 10 monclient(hunting): get_auth_request con 0x7f07940b2310 auth_method 0 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.083+0000 7f0798edb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.083+0000 7f0798edb640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.083+0000 7f0798edb640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:54.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.083+0000 7f0798edb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:54.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.083+0000 7f0798edb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:54.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.083+0000 7f0798edb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:54.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.085+0000 7f0798edb640 10 monclient(hunting): handle_auth_done global_id 21616 payload 274 2024-01-21T05:05:54.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.085+0000 7f0798edb640 1 -- 172.21.15.69:0/625855175 >> v2:172.21.15.69:3300/0 conn(0x7f07940a9d10 msgr2=0x7f079413f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:54.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.085+0000 7f0798edb640 1 --2- 172.21.15.69:0/625855175 >> v2:172.21.15.69:3300/0 conn(0x7f07940a9d10 0x7f079413f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.085+0000 7f0798edb640 1 -- 172.21.15.69:0/625855175 >> v1:172.21.15.69:6789/0 conn(0x7f07940a8b20 legacy=0x7f0794145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:54.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.085+0000 7f0798edb640 10 monclient: _finish_hunting 0 2024-01-21T05:05:54.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.085+0000 7f0798edb640 1 monclient: found mon.noname-c 2024-01-21T05:05:54.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.085+0000 7f0798edb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:54.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.085+0000 7f0798edb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T05:05:54.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.085+0000 7f0798edb640 1 -- 172.21.15.69:0/625855175 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0794146290 con 0x7f07940b2310 2024-01-21T05:05:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.087+0000 7f0798edb640 1 --2- 172.21.15.69:0/625855175 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07940b2310 0x7f079413f5c0 secure :-1 s=READY pgs=5237 cs=0 l=1 rev1=1 crypto rx=0x7f07940aa150 tx=0x7f078c02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71ba567eba21ba9d server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:54.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 1 -- 172.21.15.69:0/625855175 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c02c9f0 con 0x7f07940b2310 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 10 monclient: dump: 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:54.092 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 10 monclient: _finish_auth 0 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 10 monclient: _check_auth_tickets 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 1 -- 172.21.15.69:0/625855175 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c02cb50 con 0x7f07940b2310 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 5 monclient: authenticate success, global_id 21616 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 1 -- 172.21.15.69:0/625855175 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c0343f0 con 0x7f07940b2310 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 4 set_mon_vals no callback set 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f0792ffd640 10 monclient: dump: 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:54.093 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:54.094 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.094 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.094 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:54.094 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.094 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:54.094 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:54.094 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:54.094 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 10 monclient: shutdown 2024-01-21T05:05:54.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:54.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 1 -- 172.21.15.69:0/625855175 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07940b2310 msgr2=0x7f079413f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 1 --2- 172.21.15.69:0/625855175 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07940b2310 0x7f079413f5c0 secure :-1 s=READY pgs=5237 cs=0 l=1 rev1=1 crypto rx=0x7f07940aa150 tx=0x7f078c02be50 comp rx=0 tx=0).stop 2024-01-21T05:05:54.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 1 -- 172.21.15.69:0/625855175 shutdown_connections 2024-01-21T05:05:54.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 1 --2- 172.21.15.69:0/625855175 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07940b2310 0x7f079413f5c0 unknown :-1 s=CLOSED pgs=5237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 1 --2- 172.21.15.69:0/625855175 >> v2:172.21.15.69:3300/0 conn(0x7f07940a9d10 0x7f079413f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 1 -- 172.21.15.69:0/625855175 >> 172.21.15.69:0/625855175 conn(0x7f0794066fb0 msgr2=0x7f07940b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:54.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 1 -- 172.21.15.69:0/625855175 shutdown_connections 2024-01-21T05:05:54.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.091+0000 7f079a93e640 1 -- 172.21.15.69:0/625855175 wait complete. 2024-01-21T05:05:54.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.092+0000 7f079a93e640 10 monclient: build_initial_monmap 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.092+0000 7f079a93e640 10 monclient: monmap: 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.092+0000 7f079a93e640 1 librados: starting msgr at 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.092+0000 7f079a93e640 1 librados: starting objecter 2024-01-21T05:05:54.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.092+0000 7f079a93e640 1 Processor -- start 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 1 -- start start 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 1 librados: setting wanted keys 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 1 librados: calling monclient init 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 10 monclient: init 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07940a8b20 0x7f0794140a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 10 monclient(hunting): picked mon.noname-a con 0x7f07940a8b20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07940a9d10 0x7f0794141140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 10 monclient(hunting): picked mon.noname-b con 0x7f07940a9d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07940b2310 0x7f079413a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 10 monclient(hunting): picked mon.noname-c con 0x7f07940b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0794141680 con 0x7f07940a8b20 2024-01-21T05:05:54.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07941418f0 con 0x7f07940a9d10 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07941428b0 con 0x7f07940b2310 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.093+0000 7f079a93e640 10 monclient(hunting): authenticate will time out at 3312.434793s 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.094+0000 7f0798edb640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07940b2310 0x7f079413a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.094+0000 7f0793fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07940a8b20 0x7f0794140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f07937fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07940a9d10 0x7f0794141140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f07937fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07940a9d10 0x7f0794141140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:41474/0 (socket says 172.21.15.69:41474) 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f07937fe640 1 -- 172.21.15.69:0/3001534852 learned_addr learned my addr 172.21.15.69:0/3001534852 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f07937fe640 10 monclient(hunting): get_auth_request con 0x7f07940a9d10 auth_method 0 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f07937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f07937fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f07937fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f0798edb640 10 monclient(hunting): get_auth_request con 0x7f07940b2310 auth_method 0 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f0798edb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f0798edb640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f0798edb640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f07937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f07937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.095+0000 7f07937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.096+0000 7f0793fff640 10 monclient(hunting): get_auth_request con 0x7f07940a8b20 auth_method 0 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.096+0000 7f0793fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.096+0000 7f0793fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.096+0000 7f0793fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.096+0000 7f0798edb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.096+0000 7f0798edb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.096+0000 7f0798edb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:54.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.097+0000 7f0793fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:54.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.097+0000 7f0793fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.097+0000 7f0793fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.098+0000 7f07937fe640 10 monclient(hunting): handle_auth_done global_id 21619 payload 824 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.098+0000 7f07937fe640 1 -- 172.21.15.69:0/3001534852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07940b2310 msgr2=0x7f079413a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.098+0000 7f07937fe640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f07940b2310 0x7f079413a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.098+0000 7f07937fe640 1 -- 172.21.15.69:0/3001534852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07940a8b20 msgr2=0x7f0794140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.098+0000 7f07937fe640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f07940a8b20 0x7f0794140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.098+0000 7f07937fe640 10 monclient: _finish_hunting 0 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.098+0000 7f07937fe640 1 monclient: found mon.noname-b 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.098+0000 7f07937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.098+0000 7f07937fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.098+0000 7f07937fe640 1 -- 172.21.15.69:0/3001534852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c009b30 con 0x7f07940a9d10 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.100+0000 7f07937fe640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07940a9d10 0x7f0794141140 secure :-1 s=READY pgs=5238 cs=0 l=1 rev1=1 crypto rx=0x7f078400b480 tx=0x7f078400b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.100+0000 7f0790ff9640 10 client.?.objecter ms_handle_connect 0x7f07940a9d10 2024-01-21T05:05:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.100+0000 7f0790ff9640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 1 -- 172.21.15.69:0/3001534852 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0784010870 con 0x7f07940a9d10 2024-01-21T05:05:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 10 monclient: dump: 2024-01-21T05:05:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 10 monclient: _finish_auth 0 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 10 monclient: _check_auth_tickets 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 1 -- 172.21.15.69:0/3001534852 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07840109d0 con 0x7f07940a9d10 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 1 -- 172.21.15.69:0/3001534852 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0784010d80 con 0x7f07940a9d10 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f079a93e640 5 monclient: authenticate success, global_id 21619 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f079a93e640 10 monclient: _renew_subs 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f079a93e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f079a93e640 1 -- 172.21.15.69:0/3001534852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0794142b90 con 0x7f07940a9d10 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f0790ff9640 10 monclient: dump: 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f07927fc640 4 set_mon_vals no callback set 2024-01-21T05:05:54.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f079a93e640 10 client.21619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:54.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f079a93e640 10 monclient: _renew_subs 2024-01-21T05:05:54.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f079a93e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:54.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f079a93e640 1 -- 172.21.15.69:0/3001534852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079413a680 con 0x7f07940a9d10 2024-01-21T05:05:54.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.103+0000 7f079a93e640 1 librados: init done 2024-01-21T05:05:54.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.104+0000 7f079a93e640 10 client.21619.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:54.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.104+0000 7f079a93e640 15 client.21619.objecter _session_command_op_assign -1 1 2024-01-21T05:05:54.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.104+0000 7f079a93e640 20 client.21619.objecter _get_session osd=-1 returning homeless 2024-01-21T05:05:54.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.104+0000 7f079a93e640 10 client.21619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:54.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.104+0000 7f079a93e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:54.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.104+0000 7f079a93e640 1 -- 172.21.15.69:0/3001534852 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f07940a9150 con 0x7f07940a9d10 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.124+0000 7f0790ff9640 1 -- 172.21.15.69:0/3001534852 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f078400f3c0 con 0x7f07940a9d10 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.125+0000 7f0790ff9640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f078006c860 0x7f078006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.125+0000 7f0793fff640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f078006c860 0x7f078006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.125+0000 7f0793fff640 10 monclient: get_auth_request con 0x7f078006c860 auth_method 0 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.125+0000 7f0790ff9640 10 client.21619.objecter ms_handle_connect 0x7f078006c860 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.125+0000 7f0793fff640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f078006c860 0x7f078006ed20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f07880017f0 tx=0x7f0788009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 1 -- 172.21.15.69:0/3001534852 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f07840587b0 con 0x7f07940a9d10 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 10 client.21619.objecter ms_dispatch 0x7f0794000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 3 client.21619.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 3 client.21619.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 10 client.21619.objecter checking command 1 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] conn(0x7f0780074550 0x7f07800769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 20 client.21619.objecter _get_session s=0x7f0780073e80 osd=7 3 2024-01-21T05:05:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 20 client.21619.objecter put_session s=0x7f0780073e80 osd=7 3 2024-01-21T05:05:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 15 client.21619.objecter _session_command_op_remove -1 1 2024-01-21T05:05:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 20 client.21619.objecter _get_session s=0x7f0780073e80 osd=7 3 2024-01-21T05:05:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 20 client.21619.objecter get_session s=0x7f0780073e80 osd=7 3 2024-01-21T05:05:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 15 client.21619.objecter _session_command_op_assign 7 1 2024-01-21T05:05:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 20 client.21619.objecter put_session s=0x7f0780073e80 osd=7 4 2024-01-21T05:05:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 10 client.21619.objecter _send_command 1 2024-01-21T05:05:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 1 -- 172.21.15.69:0/3001534852 --> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07800770c0 con 0x7f0780074550 2024-01-21T05:05:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.127+0000 7f0790ff9640 20 client.21619.objecter dump_active .. 0 homeless 2024-01-21T05:05:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.128+0000 7f0798edb640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] conn(0x7f0780074550 0x7f07800769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:54.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.128+0000 7f0798edb640 10 monclient: get_auth_request con 0x7f0780074550 auth_method 0 2024-01-21T05:05:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.130+0000 7f0790ff9640 1 -- 172.21.15.69:0/3001534852 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_get_version_reply(handle=1 version=1109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0784054af0 con 0x7f07940a9d10 2024-01-21T05:05:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.130+0000 7f0790ff9640 10 monclient: handle_get_version_reply finishing 1 version 1109 2024-01-21T05:05:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.131+0000 7f0798edb640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] conn(0x7f0780074550 0x7f07800769f0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:54.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.131+0000 7f0790ff9640 10 client.21619.objecter ms_handle_connect 0x7f0780074550 2024-01-21T05:05:54.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.155+0000 7f0790ff9640 1 -- 172.21.15.69:0/3001534852 <== osd.7 v2:172.21.15.144:6810/14156089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f07800770c0 con 0x7f0780074550 2024-01-21T05:05:54.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.155+0000 7f0790ff9640 10 client.21619.objecter ms_dispatch 0x7f0794000b90 command_reply(tid 1: 0 ) v1 2024-01-21T05:05:54.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.155+0000 7f0790ff9640 10 client.21619.objecter _finish_command 1 = system:0 2024-01-21T05:05:54.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.155+0000 7f0790ff9640 20 client.21619.objecter put_session s=0x7f0780073e80 osd=7 4 2024-01-21T05:05:54.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.155+0000 7f0790ff9640 15 client.21619.objecter _session_command_op_remove 7 1 2024-01-21T05:05:54.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.168+0000 7f079a93e640 10 client.21619.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-01-21T05:05:54.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.168+0000 7f079a93e640 15 client.21619.objecter _session_command_op_assign -1 2 2024-01-21T05:05:54.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.168+0000 7f079a93e640 20 client.21619.objecter _get_session s=0x7f0780073e80 osd=7 3 2024-01-21T05:05:54.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.168+0000 7f079a93e640 20 client.21619.objecter put_session s=0x7f0780073e80 osd=7 3 2024-01-21T05:05:54.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.168+0000 7f079a93e640 20 client.21619.objecter _get_session s=0x7f0780073e80 osd=7 3 2024-01-21T05:05:54.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.168+0000 7f079a93e640 15 client.21619.objecter _session_command_op_remove -1 2 2024-01-21T05:05:54.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.168+0000 7f079a93e640 20 client.21619.objecter get_session s=0x7f0780073e80 osd=7 3 2024-01-21T05:05:54.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.168+0000 7f079a93e640 15 client.21619.objecter _session_command_op_assign 7 2 2024-01-21T05:05:54.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.168+0000 7f079a93e640 20 client.21619.objecter put_session s=0x7f0780073e80 osd=7 4 2024-01-21T05:05:54.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.168+0000 7f079a93e640 10 client.21619.objecter _send_command 2 2024-01-21T05:05:54.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.168+0000 7f079a93e640 1 -- 172.21.15.69:0/3001534852 --> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0794135a10 con 0x7f0780074550 2024-01-21T05:05:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f0790ff9640 1 -- 172.21.15.69:0/3001534852 <== osd.7 v2:172.21.15.144:6810/14156089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0794135a10 con 0x7f0780074550 2024-01-21T05:05:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f0790ff9640 10 client.21619.objecter ms_dispatch 0x7f0794000b90 command_reply(tid 2: 0 ) v1 2024-01-21T05:05:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f0790ff9640 10 client.21619.objecter _finish_command 2 = system:0 2024-01-21T05:05:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f0790ff9640 20 client.21619.objecter put_session s=0x7f0780073e80 osd=7 4 2024-01-21T05:05:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f0790ff9640 15 client.21619.objecter _session_command_op_remove 7 2 2024-01-21T05:05:54.183 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-01-21T05:05:54.183 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-21T05:05:54.183 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-01-21T05:05:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f079a93e640 10 librados: watch_flush enter 2024-01-21T05:05:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f079a93e640 10 librados: watch_flush exit 2024-01-21T05:05:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f079a93e640 10 client.21619.objecter close_session for osd.7 2024-01-21T05:05:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f079a93e640 1 -- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] conn(0x7f0780074550 msgr2=0x7f07800769f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f079a93e640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:6810/14156089,v1:172.21.15.144:6811/14156089] conn(0x7f0780074550 0x7f07800769f0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f079a93e640 20 client.21619.objecter put_session s=0x7f0780073e80 osd=7 1 2024-01-21T05:05:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f079a93e640 20 client.21619.objecter shutdown clearing up homeless session... 2024-01-21T05:05:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.182+0000 7f079a93e640 10 client.21619.objecter successfully canceled tick 2024-01-21T05:05:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f079a93e640 1 -- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f078006c860 msgr2=0x7f078006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f079a93e640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f078006c860 0x7f078006ed20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f07880017f0 tx=0x7f0788009040 comp rx=0 tx=0).stop 2024-01-21T05:05:54.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f079a93e640 10 monclient: shutdown 2024-01-21T05:05:54.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f079a93e640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:54.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f079a93e640 1 -- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07940a9d10 msgr2=0x7f0794141140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f079a93e640 1 --2- 172.21.15.69:0/3001534852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f07940a9d10 0x7f0794141140 secure :-1 s=READY pgs=5238 cs=0 l=1 rev1=1 crypto rx=0x7f078400b480 tx=0x7f078400b940 comp rx=0 tx=0).stop 2024-01-21T05:05:54.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f0798edb640 1 -- 172.21.15.69:0/3001534852 reap_dead start 2024-01-21T05:05:54.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f079a93e640 1 -- 172.21.15.69:0/3001534852 shutdown_connections 2024-01-21T05:05:54.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f079a93e640 1 -- 172.21.15.69:0/3001534852 >> 172.21.15.69:0/3001534852 conn(0x7f0794066fb0 msgr2=0x7f079413ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:54.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f079a93e640 1 -- 172.21.15.69:0/3001534852 shutdown_connections 2024-01-21T05:05:54.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f079a93e640 1 -- 172.21.15.69:0/3001534852 wait complete. 2024-01-21T05:05:54.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.183+0000 7f079a93e640 1 librados: shutdown 2024-01-21T05:05:54.195 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-21T05:05:54.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.297+0000 7f632dda9640 10 monclient: get_monmap_and_config 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.297+0000 7f632dda9640 10 monclient: build_initial_monmap 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.297+0000 7f632dda9640 10 monclient: monmap: 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:54.298219+0000 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:54.298219+0000 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:54.298 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:54.299 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.297+0000 7f632dda9640 1 Processor -- start 2024-01-21T05:05:54.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 1 -- start start 2024-01-21T05:05:54.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 10 monclient: init 2024-01-21T05:05:54.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:54.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63280a8b20 addr v1:172.21.15.144:6789/0 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f63280a9d10 0x7f632813f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 10 monclient(hunting): picked mon.noname-c con 0x7f63280a9d10 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63280b2310 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63281456f0 con 0x7f63280b2310 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 1 -- --> v1:172.21.15.144:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63280a98c0 con 0x7f63280a8b20 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6328146290 con 0x7f63280a9d10 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.298+0000 7f632dda9640 10 monclient(hunting): authenticate will time out at 3312.639856s 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.299+0000 7f6327fff640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f63280b2310 0x7f632813f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50270/0 (socket says 172.21.15.69:50270) 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.299+0000 7f6327fff640 1 -- 172.21.15.69:0/982299236 learned_addr learned my addr 172.21.15.69:0/982299236 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.299+0000 7f6326ffd640 1 --2- 172.21.15.69:0/982299236 >> v2:172.21.15.144:3300/0 conn(0x7f63280a9d10 0x7f632813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:54.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.299+0000 7f6326ffd640 10 monclient(hunting): get_auth_request con 0x7f63280a9d10 auth_method 0 2024-01-21T05:05:54.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.299+0000 7f6326ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:54.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.299+0000 7f6326ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:54.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.299+0000 7f6326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.300+0000 7f6326ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.300+0000 7f6326ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.300+0000 7f6326ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.301+0000 7f63267fc640 1 -- 172.21.15.69:0/982299236 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772840996 0 0) 0x7f63281456f0 con 0x7f63280b2310 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.301+0000 7f63267fc640 10 monclient(hunting): _negotiate 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.301+0000 7f63267fc640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.301+0000 7f63267fc640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.301+0000 7f63267fc640 10 monclient(hunting): my global_id is 17514 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.301+0000 7f63267fc640 1 -- 172.21.15.69:0/982299236 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6310003520 con 0x7f63280b2310 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.302+0000 7f6326ffd640 10 monclient(hunting): handle_auth_done global_id 21622 payload 274 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.302+0000 7f6326ffd640 1 -- 172.21.15.69:0/982299236 >> v1:172.21.15.144:6789/0 conn(0x7f63280a8b20 legacy=0x7f6328145b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.302+0000 7f6326ffd640 1 -- 172.21.15.69:0/982299236 >> v1:172.21.15.69:6789/0 conn(0x7f63280b2310 legacy=0x7f632813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.302+0000 7f6326ffd640 10 monclient: _finish_hunting 0 2024-01-21T05:05:54.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.302+0000 7f6326ffd640 1 monclient: found mon.noname-c 2024-01-21T05:05:54.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.302+0000 7f6326ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:54.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.302+0000 7f6326ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T05:05:54.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.302+0000 7f6326ffd640 1 -- 172.21.15.69:0/982299236 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63281463d0 con 0x7f63280a9d10 2024-01-21T05:05:54.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.305+0000 7f6326ffd640 1 --2- 172.21.15.69:0/982299236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63280a9d10 0x7f632813f080 secure :-1 s=READY pgs=5240 cs=0 l=1 rev1=1 crypto rx=0x7f631c0097a0 tx=0x7f631c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f4a4c2ef078711c server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:54.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 1 -- 172.21.15.69:0/982299236 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c030610 con 0x7f63280a9d10 2024-01-21T05:05:54.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:54.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:05:54.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 10 monclient: dump: 2024-01-21T05:05:54.310 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:54.310 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:54.310 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.310 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.310 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:54.310 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.310 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:54.310 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 10 monclient: _finish_auth 0 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 10 monclient: _check_auth_tickets 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 1 -- 172.21.15.69:0/982299236 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c030770 con 0x7f63280a9d10 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 5 monclient: authenticate success, global_id 21622 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 1 -- 172.21.15.69:0/982299236 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c030b00 con 0x7f63280a9d10 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 4 set_mon_vals no callback set 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:54.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f63267fc640 10 monclient: dump: 2024-01-21T05:05:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:54.312 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:54.312 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 10 monclient: shutdown 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 1 -- 172.21.15.69:0/982299236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63280a9d10 msgr2=0x7f632813f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 1 --2- 172.21.15.69:0/982299236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63280a9d10 0x7f632813f080 secure :-1 s=READY pgs=5240 cs=0 l=1 rev1=1 crypto rx=0x7f631c0097a0 tx=0x7f631c02b960 comp rx=0 tx=0).stop 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 1 -- 172.21.15.69:0/982299236 shutdown_connections 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 1 --2- 172.21.15.69:0/982299236 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63280a9d10 0x7f632813f080 unknown :-1 s=CLOSED pgs=5240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 1 -- 172.21.15.69:0/982299236 >> 172.21.15.69:0/982299236 conn(0x7f6328066fb0 msgr2=0x7f63280b7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 1 -- 172.21.15.69:0/982299236 shutdown_connections 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.309+0000 7f632dda9640 1 -- 172.21.15.69:0/982299236 wait complete. 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.310+0000 7f632dda9640 10 monclient: build_initial_monmap 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.310+0000 7f632dda9640 10 monclient: monmap: 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:54.316 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.310+0000 7f632dda9640 1 librados: starting msgr at 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.310+0000 7f632dda9640 1 librados: starting objecter 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.310+0000 7f632dda9640 1 Processor -- start 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 1 -- start start 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 1 librados: setting wanted keys 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 1 librados: calling monclient init 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 10 monclient: init 2024-01-21T05:05:54.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 10 monclient: _add_conns ranks=[0,1,2] 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63280a8b20 0x7f632813b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 10 monclient(hunting): picked mon.noname-a con 0x7f63280a8b20 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63280a9d10 0x7f6328138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 10 monclient(hunting): picked mon.noname-b con 0x7f63280a9d10 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63280b2310 0x7f63281393f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 10 monclient(hunting): picked mon.noname-c con 0x7f63280b2310 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6328142860 con 0x7f63280a8b20 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632813a570 con 0x7f63280a9d10 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632813a7e0 con 0x7f63280b2310 2024-01-21T05:05:54.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.311+0000 7f632dda9640 10 monclient(hunting): authenticate will time out at 3312.652845s 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f63277fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63280a8b20 0x7f632813b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6326ffd640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63280a9d10 0x7f6328138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6327fff640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63280b2310 0x7f63281393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f63277fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63280a8b20 0x7f632813b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41852/0 (socket says 172.21.15.69:41852) 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f63277fe640 1 -- 172.21.15.69:0/426284821 learned_addr learned my addr 172.21.15.69:0/426284821 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f63277fe640 10 monclient(hunting): get_auth_request con 0x7f63280a8b20 auth_method 0 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f63277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f63277fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f63277fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6326ffd640 10 monclient(hunting): get_auth_request con 0x7f63280a9d10 auth_method 0 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6326ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6326ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:54.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6327fff640 10 monclient(hunting): get_auth_request con 0x7f63280b2310 auth_method 0 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6327fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6327fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6327fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6326ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6326ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f6326ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f63277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.312+0000 7f63277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.313+0000 7f63277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.313+0000 7f6327fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.313+0000 7f6327fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:54.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.313+0000 7f6327fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:54.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.315+0000 7f63277fe640 10 monclient(hunting): handle_auth_done global_id 17517 payload 824 2024-01-21T05:05:54.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.315+0000 7f63277fe640 1 -- 172.21.15.69:0/426284821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63280b2310 msgr2=0x7f63281393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.315+0000 7f63277fe640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63280b2310 0x7f63281393f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.315+0000 7f63277fe640 1 -- 172.21.15.69:0/426284821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63280a9d10 msgr2=0x7f6328138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.315+0000 7f63277fe640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63280a9d10 0x7f6328138eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.315+0000 7f63277fe640 10 monclient: _finish_hunting 0 2024-01-21T05:05:54.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.315+0000 7f63277fe640 1 monclient: found mon.noname-a 2024-01-21T05:05:54.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.315+0000 7f63277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:54.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.315+0000 7f63277fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:05:54.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.315+0000 7f63277fe640 1 -- 172.21.15.69:0/426284821 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631c009400 con 0x7f63280a8b20 2024-01-21T05:05:54.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.326+0000 7f63277fe640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63280a8b20 0x7f632813b4a0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7f631800c430 tx=0x7f631800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:54.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.326+0000 7f630ffff640 10 client.?.objecter ms_handle_connect 0x7f63280a8b20 2024-01-21T05:05:54.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.326+0000 7f630ffff640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:54.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 1 -- 172.21.15.69:0/426284821 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63180052d0 con 0x7f63280a8b20 2024-01-21T05:05:54.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:54.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 10 monclient: dump: 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 10 monclient: _finish_auth 0 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 10 monclient: _check_auth_tickets 2024-01-21T05:05:54.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 1 -- 172.21.15.69:0/426284821 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6318005450 con 0x7f63280a8b20 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 1 -- 172.21.15.69:0/426284821 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6318007260 con 0x7f63280a8b20 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f630ffff640 10 monclient: dump: 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:54.341 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:54.342 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:54.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f632dda9640 5 monclient: authenticate success, global_id 17517 2024-01-21T05:05:54.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f632dda9640 10 monclient: _renew_subs 2024-01-21T05:05:54.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f632dda9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:54.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f632dda9640 1 -- 172.21.15.69:0/426284821 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632813aac0 con 0x7f63280a8b20 2024-01-21T05:05:54.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f6325ffb640 4 set_mon_vals no callback set 2024-01-21T05:05:54.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f632dda9640 10 client.17517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:54.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f632dda9640 10 monclient: _renew_subs 2024-01-21T05:05:54.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f632dda9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:54.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f632dda9640 1 -- 172.21.15.69:0/426284821 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63281423f0 con 0x7f63280a8b20 2024-01-21T05:05:54.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.338+0000 7f632dda9640 1 librados: init done 2024-01-21T05:05:54.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.339+0000 7f632dda9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:54.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.339+0000 7f632dda9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:54.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.339+0000 7f632dda9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.69:3300/0 2024-01-21T05:05:54.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.339+0000 7f632dda9640 1 -- 172.21.15.69:0/426284821 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63280b1a30 con 0x7f63280a8b20 2024-01-21T05:05:54.390 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:22.297 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:54.436 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:22.311 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:54.437 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:22.344 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:54.438 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:22.313 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:54.438 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:22.345 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:54.448 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:22.355 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:54.449 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:22.324 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:54.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.660+0000 7f630ffff640 1 -- 172.21.15.69:0/426284821 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f63180073c0 con 0x7f63280a8b20 2024-01-21T05:05:54.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.660+0000 7f630ffff640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f631406c8b0 0x7f631406ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.660+0000 7f6326ffd640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f631406c8b0 0x7f631406ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.661+0000 7f6326ffd640 10 monclient: get_auth_request con 0x7f631406c8b0 auth_method 0 2024-01-21T05:05:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.661+0000 7f630ffff640 10 client.17517.objecter ms_handle_connect 0x7f631406c8b0 2024-01-21T05:05:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.661+0000 7f6326ffd640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f631406c8b0 0x7f631406ed70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f631c036000 tx=0x7f631c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:54.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.674+0000 7f630ffff640 1 -- 172.21.15.69:0/426284821 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f6318004f10 con 0x7f63280a8b20 2024-01-21T05:05:54.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.674+0000 7f630ffff640 10 client.17517.objecter ms_dispatch 0x7f6328000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:54.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.674+0000 7f630ffff640 3 client.17517.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:54.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.674+0000 7f630ffff640 3 client.17517.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:54.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.674+0000 7f630ffff640 20 client.17517.objecter dump_active .. 0 homeless 2024-01-21T05:05:54.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.674+0000 7f630ffff640 1 -- 172.21.15.69:0/426284821 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f6318053e70 con 0x7f63280a8b20 2024-01-21T05:05:54.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.674+0000 7f630ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:54.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.674+0000 7f630ffff640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:54.678 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:22.585 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:54.744 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:22.651 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:05:54.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.904+0000 7f632dda9640 1 -- 172.21.15.69:0/426284821 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f63280a9150 con 0x7f631406c8b0 2024-01-21T05:05:54.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.905+0000 7f630ffff640 1 -- 172.21.15.69:0/426284821 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f631c0308e0 con 0x7f631406c8b0 2024-01-21T05:05:54.905 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 7 to deep-scrub 2024-01-21T05:05:54.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 10 librados: watch_flush enter 2024-01-21T05:05:54.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 10 librados: watch_flush exit 2024-01-21T05:05:54.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 20 client.17517.objecter shutdown clearing up homeless session... 2024-01-21T05:05:54.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 10 client.17517.objecter successfully canceled tick 2024-01-21T05:05:54.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 1 -- 172.21.15.69:0/426284821 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f631406c8b0 msgr2=0x7f631406ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f631406c8b0 0x7f631406ed70 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f631c036000 tx=0x7f631c008030 comp rx=0 tx=0).stop 2024-01-21T05:05:54.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 10 monclient: shutdown 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 1 -- 172.21.15.69:0/426284821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63280a8b20 msgr2=0x7f632813b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63280a8b20 0x7f632813b4a0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7f631800c430 tx=0x7f631800c8f0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 1 -- 172.21.15.69:0/426284821 shutdown_connections 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f631406c8b0 0x7f631406ed70 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f63280b2310 0x7f63281393f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63280a9d10 0x7f6328138eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.906+0000 7f632dda9640 1 --2- 172.21.15.69:0/426284821 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63280a8b20 0x7f632813b4a0 unknown :-1 s=CLOSED pgs=3508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.907+0000 7f632dda9640 1 -- 172.21.15.69:0/426284821 >> 172.21.15.69:0/426284821 conn(0x7f6328066fb0 msgr2=0x7f6328145b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.907+0000 7f632dda9640 1 -- 172.21.15.69:0/426284821 shutdown_connections 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.907+0000 7f632dda9640 1 -- 172.21.15.69:0/426284821 wait complete. 2024-01-21T05:05:54.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:54.907+0000 7f632dda9640 1 librados: shutdown 2024-01-21T05:05:54.919 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T05:05:55.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.033+0000 7f2aa4b25640 10 monclient: get_monmap_and_config 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.033+0000 7f2aa4b25640 10 monclient: build_initial_monmap 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.033+0000 7f2aa4b25640 10 monclient: monmap: 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:05:55.034061+0000 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:05:55.034061+0000 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:05:55.034 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:05:55.035 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:55.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.033+0000 7f2aa4b25640 1 Processor -- start 2024-01-21T05:05:55.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.033+0000 7f2aa4b25640 1 -- start start 2024-01-21T05:05:55.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.033+0000 7f2aa4b25640 10 monclient: init 2024-01-21T05:05:55.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:55.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2aa00a8a40 0x7f2aa0145b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 10 monclient(hunting): picked mon.noname-a con 0x7f2aa00a8a40 addr v2:172.21.15.69:3300/0 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2aa01460a0 0x7f2aa013f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 10 monclient(hunting): picked mon.noname-c con 0x7f2aa01460a0 addr v2:172.21.15.144:3300/0 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2aa00b21c0 addr v1:172.21.15.69:6789/0 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa00aa9b0 con 0x7f2aa00b21c0 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00aa7d0 con 0x7f2aa00a8a40 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00aac90 con 0x7f2aa01460a0 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.034+0000 7f2aa4b25640 10 monclient(hunting): authenticate will time out at 3313.375857s 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.035+0000 7f2a97fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2aa01460a0 0x7f2aa013f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.035+0000 7f2a97fff640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2aa01460a0 0x7f2aa013f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:39100/0 (socket says 172.21.15.69:39100) 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.035+0000 7f2a97fff640 1 -- 172.21.15.69:0/4058776509 learned_addr learned my addr 172.21.15.69:0/4058776509 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:55.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.035+0000 7f2a97fff640 10 monclient(hunting): get_auth_request con 0x7f2aa01460a0 auth_method 0 2024-01-21T05:05:55.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.035+0000 7f2a97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:55.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.035+0000 7f2a97fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:55.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.035+0000 7f2a97fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:55.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.036+0000 7f2a97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:55.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.036+0000 7f2a97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:55.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.036+0000 7f2a97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:55.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.037+0000 7f2a97fff640 10 monclient(hunting): handle_auth_done global_id 21628 payload 274 2024-01-21T05:05:55.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.037+0000 7f2a97fff640 1 -- 172.21.15.69:0/4058776509 >> v2:172.21.15.69:3300/0 conn(0x7f2aa00a8a40 msgr2=0x7f2aa0145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:55.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.037+0000 7f2a97fff640 1 --2- 172.21.15.69:0/4058776509 >> v2:172.21.15.69:3300/0 conn(0x7f2aa00a8a40 0x7f2aa0145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:55.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.037+0000 7f2a97fff640 1 -- 172.21.15.69:0/4058776509 >> v1:172.21.15.69:6789/0 conn(0x7f2aa00b21c0 legacy=0x7f2aa0141f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:05:55.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.037+0000 7f2a97fff640 10 monclient: _finish_hunting 0 2024-01-21T05:05:55.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.037+0000 7f2a97fff640 1 monclient: found mon.noname-c 2024-01-21T05:05:55.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.037+0000 7f2a97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:55.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.037+0000 7f2a97fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T05:05:55.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.037+0000 7f2a97fff640 1 -- 172.21.15.69:0/4058776509 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa00aaf70 con 0x7f2aa01460a0 2024-01-21T05:05:55.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.039+0000 7f2a97fff640 1 --2- 172.21.15.69:0/4058776509 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2aa01460a0 0x7f2aa013f1a0 secure :-1 s=READY pgs=5241 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00aa2a0 tx=0x7f2a8802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adfd1c9e50d220fa server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:55.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 1 -- 172.21.15.69:0/4058776509 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a88030860 con 0x7f2aa01460a0 2024-01-21T05:05:55.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 10 monclient: dump: 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 10 monclient: _finish_auth 0 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 10 monclient: _check_auth_tickets 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 1 -- 172.21.15.69:0/4058776509 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a880309c0 con 0x7f2aa01460a0 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 5 monclient: authenticate success, global_id 21628 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 1 -- 172.21.15.69:0/4058776509 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a88030d50 con 0x7f2aa01460a0 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 10 monclient: get_monmap_and_config success 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 4 set_mon_vals no callback set 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2a9dd9b640 10 monclient: dump: 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:55.046 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:55.046 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:55.046 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:55.046 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:55.046 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:55.046 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:55.046 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 10 monclient: shutdown 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 1 -- 172.21.15.69:0/4058776509 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2aa01460a0 msgr2=0x7f2aa013f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 1 --2- 172.21.15.69:0/4058776509 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2aa01460a0 0x7f2aa013f1a0 secure :-1 s=READY pgs=5241 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00aa2a0 tx=0x7f2a8802b9a0 comp rx=0 tx=0).stop 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 1 -- 172.21.15.69:0/4058776509 shutdown_connections 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 1 --2- 172.21.15.69:0/4058776509 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2aa01460a0 0x7f2aa013f1a0 unknown :-1 s=CLOSED pgs=5241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 1 --2- 172.21.15.69:0/4058776509 >> v2:172.21.15.69:3300/0 conn(0x7f2aa00a8a40 0x7f2aa0145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 1 -- 172.21.15.69:0/4058776509 >> 172.21.15.69:0/4058776509 conn(0x7f2aa0066f50 msgr2=0x7f2aa00b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 1 -- 172.21.15.69:0/4058776509 shutdown_connections 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.043+0000 7f2aa4b25640 1 -- 172.21.15.69:0/4058776509 wait complete. 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.044+0000 7f2aa4b25640 10 monclient: build_initial_monmap 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.044+0000 7f2aa4b25640 10 monclient: monmap: 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:05:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.044+0000 7f2aa4b25640 1 librados: starting msgr at 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.044+0000 7f2aa4b25640 1 librados: starting objecter 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 1 Processor -- start 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 1 -- start start 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 1 librados: setting wanted keys 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 1 librados: calling monclient init 2024-01-21T05:05:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 10 monclient: init 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 10 monclient: _reopen_session rank -1 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 10 monclient: _add_conns ranks=[2,0,1] 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00a8a40 0x7f2aa0144c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 10 monclient(hunting): picked mon.noname-c con 0x7f2aa00a8a40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa00b21c0 0x7f2aa01424f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 10 monclient(hunting): picked mon.noname-a con 0x7f2aa00b21c0 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2aa01460a0 0x7f2aa01d7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 10 monclient(hunting): picked mon.noname-b con 0x7f2aa01460a0 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00b6330 con 0x7f2aa00b21c0 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa0142bc0 con 0x7f2aa01460a0 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 10 monclient(hunting): start opening mon connection 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00b53b0 con 0x7f2aa00a8a40 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 10 monclient(hunting): _renew_subs 2024-01-21T05:05:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.045+0000 7f2aa4b25640 10 monclient(hunting): authenticate will time out at 3313.386949s 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9e59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00a8a40 0x7f2aa0144c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9ed9d640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2aa01460a0 0x7f2aa01d7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9e59c640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00a8a40 0x7f2aa0144c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:36952/0 (socket says 172.21.15.69:36952) 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9e59c640 1 -- 172.21.15.69:0/3293727133 learned_addr learned my addr 172.21.15.69:0/3293727133 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9e59c640 10 monclient(hunting): get_auth_request con 0x7f2aa00a8a40 auth_method 0 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9e59c640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9e59c640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9ed9d640 10 monclient(hunting): get_auth_request con 0x7f2aa01460a0 auth_method 0 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9ed9d640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.046+0000 7f2a9ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.047+0000 7f2a9ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.047+0000 7f2a9ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:55.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.047+0000 7f2a9ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:55.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.047+0000 7f2a9e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:05:55.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.047+0000 7f2a9e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:05:55.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.047+0000 7f2a9e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.050+0000 7f2a9ed9d640 10 monclient(hunting): handle_auth_done global_id 21631 payload 824 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.050+0000 7f2a9ed9d640 1 -- 172.21.15.69:0/3293727133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00a8a40 msgr2=0x7f2aa0144c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.050+0000 7f2a9ed9d640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00a8a40 0x7f2aa0144c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.050+0000 7f2a9ed9d640 1 -- 172.21.15.69:0/3293727133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa00b21c0 msgr2=0x7f2aa01424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.050+0000 7f2a9ed9d640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa00b21c0 0x7f2aa01424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.050+0000 7f2a9ed9d640 10 monclient: _finish_hunting 0 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.050+0000 7f2a9ed9d640 1 monclient: found mon.noname-b 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.050+0000 7f2a9ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.050+0000 7f2a9ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.050+0000 7f2a9ed9d640 1 -- 172.21.15.69:0/3293727133 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a880094d0 con 0x7f2aa01460a0 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.052+0000 7f2a9ed9d640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2aa01460a0 0x7f2aa01d7370 secure :-1 s=READY pgs=5242 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800c780 tx=0x7f2a9800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.052+0000 7f2a96ffd640 10 client.?.objecter ms_handle_connect 0x7f2aa01460a0 2024-01-21T05:05:55.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.052+0000 7f2a96ffd640 10 client.?.objecter resend_mon_ops 2024-01-21T05:05:55.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 1 -- 172.21.15.69:0/3293727133 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98003b70 con 0x7f2aa01460a0 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 10 monclient: dump: 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 10 monclient: _finish_auth 0 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 10 monclient: _check_auth_tickets 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 1 -- 172.21.15.69:0/3293727133 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a98003cf0 con 0x7f2aa01460a0 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2aa4b25640 5 monclient: authenticate success, global_id 21631 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2aa4b25640 10 monclient: _renew_subs 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2aa4b25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2aa4b25640 1 -- 172.21.15.69:0/3293727133 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa0141c90 con 0x7f2aa01460a0 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 1 -- 172.21.15.69:0/3293727133 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98003ea0 con 0x7f2aa01460a0 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a96ffd640 10 monclient: dump: 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2a9d59a640 4 set_mon_vals no callback set 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2aa4b25640 10 client.21631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2aa4b25640 10 monclient: _renew_subs 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2aa4b25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2aa4b25640 1 -- 172.21.15.69:0/3293727133 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa01421c0 con 0x7f2aa01460a0 2024-01-21T05:05:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.114+0000 7f2aa4b25640 1 librados: init done 2024-01-21T05:05:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.115+0000 7f2aa4b25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.115+0000 7f2aa4b25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.115+0000 7f2aa4b25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:05:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.115+0000 7f2aa4b25640 1 -- 172.21.15.69:0/3293727133 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa0142a30 con 0x7f2aa01460a0 2024-01-21T05:05:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.371+0000 7f2a96ffd640 1 -- 172.21.15.69:0/3293727133 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2a980052b0 con 0x7f2aa01460a0 2024-01-21T05:05:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.371+0000 7f2a96ffd640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2a8006c890 0x7f2a8006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:05:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.372+0000 7f2a9e59c640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2a8006c890 0x7f2a8006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:05:55.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.372+0000 7f2a9e59c640 10 monclient: get_auth_request con 0x7f2a8006c890 auth_method 0 2024-01-21T05:05:55.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.372+0000 7f2a9e59c640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2a8006c890 0x7f2a8006ed50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009a40 tx=0x7f2a900091c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:05:55.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.372+0000 7f2a96ffd640 10 client.21631.objecter ms_handle_connect 0x7f2a8006c890 2024-01-21T05:05:55.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.374+0000 7f2a96ffd640 1 -- 172.21.15.69:0/3293727133 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f2a98057e90 con 0x7f2aa01460a0 2024-01-21T05:05:55.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.374+0000 7f2a96ffd640 10 client.21631.objecter ms_dispatch 0x7f2aa0000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:05:55.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.374+0000 7f2a96ffd640 3 client.21631.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:05:55.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.374+0000 7f2a96ffd640 3 client.21631.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:05:55.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.375+0000 7f2a96ffd640 20 client.21631.objecter dump_active .. 0 homeless 2024-01-21T05:05:55.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.385+0000 7f2a96ffd640 1 -- 172.21.15.69:0/3293727133 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2a980541d0 con 0x7f2aa01460a0 2024-01-21T05:05:55.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.385+0000 7f2a96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:05:55.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.385+0000 7f2a96ffd640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:05:55.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.561+0000 7f2aa4b25640 1 -- 172.21.15.69:0/3293727133 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2aa00b64a0 con 0x7f2a8006c890 2024-01-21T05:05:55.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.565+0000 7f2a96ffd640 1 -- 172.21.15.69:0/3293727133 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151351 (secure 0 0 0) 0x7f2aa00b64a0 con 0x7f2a8006c890 2024-01-21T05:05:55.568 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T05:05:55.569 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-01-21T05:05:55.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 10 librados: watch_flush enter 2024-01-21T05:05:55.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 10 librados: watch_flush exit 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 20 client.21631.objecter shutdown clearing up homeless session... 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 10 client.21631.objecter successfully canceled tick 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 -- 172.21.15.69:0/3293727133 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2a8006c890 msgr2=0x7f2a8006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2a8006c890 0x7f2a8006ed50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009a40 tx=0x7f2a900091c0 comp rx=0 tx=0).stop 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 10 monclient: shutdown 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 -- 172.21.15.69:0/3293727133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2aa01460a0 msgr2=0x7f2aa01d7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2aa01460a0 0x7f2aa01d7370 secure :-1 s=READY pgs=5242 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800c780 tx=0x7f2a9800cc40 comp rx=0 tx=0).stop 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 -- 172.21.15.69:0/3293727133 shutdown_connections 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2a8006c890 0x7f2a8006ed50 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2aa01460a0 0x7f2aa01d7370 unknown :-1 s=CLOSED pgs=5242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2aa00b21c0 0x7f2aa01424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:55.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 --2- 172.21.15.69:0/3293727133 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2aa00a8a40 0x7f2aa0144c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:05:55.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 -- 172.21.15.69:0/3293727133 >> 172.21.15.69:0/3293727133 conn(0x7f2aa0066f50 msgr2=0x7f2aa00b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:05:55.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 -- 172.21.15.69:0/3293727133 shutdown_connections 2024-01-21T05:05:55.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 -- 172.21.15.69:0/3293727133 wait complete. 2024-01-21T05:05:55.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:05:55.568+0000 7f2aa4b25640 1 librados: shutdown 2024-01-21T05:05:55.582 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":2084,"stamp":"2024-01-21T05:05:54.035571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":607824,"kb_used_data":88984,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754366896,"statfs":{"total":773094113280,"available":772471701504,"internally_reserved":0,"allocated":91119616,"data_stored":49865650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":529714736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":133,"apply_latency_ms":133,"commit_latency_ns":133000000,"apply_latency_ns":133000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001820"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":1047,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.491634+0000","last_change":"2024-01-21T04:34:59.893360+0000","last_active":"2024-01-21T05:05:26.491634+0000","last_peered":"2024-01-21T05:05:26.491634+0000","last_clean":"2024-01-21T05:05:26.491634+0000","last_became_active":"2024-01-21T04:34:59.874210+0000","last_became_peered":"2024-01-21T04:34:59.874210+0000","last_unstale":"2024-01-21T05:05:26.491634+0000","last_undegraded":"2024-01-21T05:05:26.491634+0000","last_fullsized":"2024-01-21T05:05:26.491634+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:03:39.790631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1064,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.373474+0000","last_change":"2024-01-21T04:35:19.549398+0000","last_active":"2024-01-21T05:05:26.373474+0000","last_peered":"2024-01-21T05:05:26.373474+0000","last_clean":"2024-01-21T05:05:26.373474+0000","last_became_active":"2024-01-21T04:35:19.542355+0000","last_became_peered":"2024-01-21T04:35:19.542355+0000","last_unstale":"2024-01-21T05:05:26.373474+0000","last_undegraded":"2024-01-21T05:05:26.373474+0000","last_fullsized":"2024-01-21T05:05:26.373474+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:17:21.163426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1088,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.371230+0000","last_change":"2024-01-21T04:35:21.254053+0000","last_active":"2024-01-21T05:05:44.371230+0000","last_peered":"2024-01-21T05:05:44.371230+0000","last_clean":"2024-01-21T05:05:44.371230+0000","last_became_active":"2024-01-21T04:35:21.234113+0000","last_became_peered":"2024-01-21T04:35:21.234113+0000","last_unstale":"2024-01-21T05:05:44.371230+0000","last_undegraded":"2024-01-21T05:05:44.371230+0000","last_fullsized":"2024-01-21T05:05:44.371230+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1052,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.596904+0000","last_change":"2024-01-21T04:35:23.469523+0000","last_active":"2024-01-21T05:05:44.596904+0000","last_peered":"2024-01-21T05:05:44.596904+0000","last_clean":"2024-01-21T05:05:44.596904+0000","last_became_active":"2024-01-21T04:35:23.463341+0000","last_became_peered":"2024-01-21T04:35:23.463341+0000","last_unstale":"2024-01-21T05:05:44.596904+0000","last_undegraded":"2024-01-21T05:05:44.596904+0000","last_fullsized":"2024-01-21T05:05:44.596904+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:40:34.277268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1053,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.673041+0000","last_change":"2024-01-21T04:35:16.948777+0000","last_active":"2024-01-21T05:05:44.673041+0000","last_peered":"2024-01-21T05:05:44.673041+0000","last_clean":"2024-01-21T05:05:44.673041+0000","last_became_active":"2024-01-21T04:35:16.942042+0000","last_became_peered":"2024-01-21T04:35:16.942042+0000","last_unstale":"2024-01-21T05:05:44.673041+0000","last_undegraded":"2024-01-21T05:05:44.673041+0000","last_fullsized":"2024-01-21T05:05:44.673041+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"116'7","reported_seq":1079,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.776703+0000","last_change":"2024-01-21T04:35:25.379657+0000","last_active":"2024-01-21T05:05:44.776703+0000","last_peered":"2024-01-21T05:05:44.776703+0000","last_clean":"2024-01-21T05:05:44.776703+0000","last_became_active":"2024-01-21T04:35:25.364561+0000","last_became_peered":"2024-01-21T04:35:25.364561+0000","last_unstale":"2024-01-21T05:05:44.776703+0000","last_undegraded":"2024-01-21T05:05:44.776703+0000","last_fullsized":"2024-01-21T05:05:44.776703+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1095,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:50.977273+0000","last_change":"2024-01-21T05:05:50.978117+0000","last_active":"2024-01-21T05:05:50.977273+0000","last_peered":"2024-01-21T05:05:50.977273+0000","last_clean":"2024-01-21T05:05:50.977273+0000","last_became_active":"2024-01-21T04:34:53.465163+0000","last_became_peered":"2024-01-21T04:34:53.465163+0000","last_unstale":"2024-01-21T05:05:50.977273+0000","last_undegraded":"2024-01-21T05:05:50.977273+0000","last_fullsized":"2024-01-21T05:05:50.977273+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:50.975781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:50.975781+0000","last_clean_scrub_stamp":"2024-01-21T05:05:50.975781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:24:33.611723+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1050,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:43.832568+0000","last_change":"2024-01-21T04:35:02.604107+0000","last_active":"2024-01-21T05:05:43.832568+0000","last_peered":"2024-01-21T05:05:43.832568+0000","last_clean":"2024-01-21T05:05:43.832568+0000","last_became_active":"2024-01-21T04:35:02.595993+0000","last_became_peered":"2024-01-21T04:35:02.595993+0000","last_unstale":"2024-01-21T05:05:43.832568+0000","last_undegraded":"2024-01-21T05:05:43.832568+0000","last_fullsized":"2024-01-21T05:05:43.832568+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:05:40.627686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1087,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.366478+0000","last_change":"2024-01-21T04:35:21.226772+0000","last_active":"2024-01-21T05:05:44.366478+0000","last_peered":"2024-01-21T05:05:44.366478+0000","last_clean":"2024-01-21T05:05:44.366478+0000","last_became_active":"2024-01-21T04:35:21.218816+0000","last_became_peered":"2024-01-21T04:35:21.218816+0000","last_unstale":"2024-01-21T05:05:44.366478+0000","last_undegraded":"2024-01-21T05:05:44.366478+0000","last_fullsized":"2024-01-21T05:05:44.366478+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1059,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:49.049388+0000","last_change":"2024-01-21T05:05:49.050212+0000","last_active":"2024-01-21T05:05:49.049388+0000","last_peered":"2024-01-21T05:05:49.049388+0000","last_clean":"2024-01-21T05:05:49.049388+0000","last_became_active":"2024-01-21T04:34:53.056955+0000","last_became_peered":"2024-01-21T04:34:53.056955+0000","last_unstale":"2024-01-21T05:05:49.049388+0000","last_undegraded":"2024-01-21T05:05:49.049388+0000","last_fullsized":"2024-01-21T05:05:49.049388+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:49.047906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:49.047906+0000","last_clean_scrub_stamp":"2024-01-21T05:05:49.047906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:41:23.348783+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1066,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.379720+0000","last_change":"2024-01-21T04:35:00.848090+0000","last_active":"2024-01-21T05:05:26.379720+0000","last_peered":"2024-01-21T05:05:26.379720+0000","last_clean":"2024-01-21T05:05:26.379720+0000","last_became_active":"2024-01-21T04:35:00.838638+0000","last_became_peered":"2024-01-21T04:35:00.838638+0000","last_unstale":"2024-01-21T05:05:26.379720+0000","last_undegraded":"2024-01-21T05:05:26.379720+0000","last_fullsized":"2024-01-21T05:05:26.379720+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:25:11.256572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1047,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:27.034452+0000","last_change":"2024-01-21T04:35:28.444122+0000","last_active":"2024-01-21T05:05:27.034452+0000","last_peered":"2024-01-21T05:05:27.034452+0000","last_clean":"2024-01-21T05:05:27.034452+0000","last_became_active":"2024-01-21T04:35:28.438809+0000","last_became_peered":"2024-01-21T04:35:28.438809+0000","last_unstale":"2024-01-21T05:05:27.034452+0000","last_undegraded":"2024-01-21T05:05:27.034452+0000","last_fullsized":"2024-01-21T05:05:27.034452+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:27:38.339736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1051,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.549014+0000","last_change":"2024-01-21T04:35:00.835300+0000","last_active":"2024-01-21T05:05:44.549014+0000","last_peered":"2024-01-21T05:05:44.549014+0000","last_clean":"2024-01-21T05:05:44.549014+0000","last_became_active":"2024-01-21T04:35:00.824479+0000","last_became_peered":"2024-01-21T04:35:00.824479+0000","last_unstale":"2024-01-21T05:05:44.549014+0000","last_undegraded":"2024-01-21T05:05:44.549014+0000","last_fullsized":"2024-01-21T05:05:44.549014+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:11:26.198997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"1109'78","reported_seq":1192,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:50.165518+0000","last_change":"2024-01-21T05:05:50.166955+0000","last_active":"2024-01-21T05:05:50.165518+0000","last_peered":"2024-01-21T05:05:50.165518+0000","last_clean":"2024-01-21T05:05:50.165518+0000","last_became_active":"2024-01-21T04:33:33.133732+0000","last_became_peered":"2024-01-21T04:33:33.133732+0000","last_unstale":"2024-01-21T05:05:50.165518+0000","last_undegraded":"2024-01-21T05:05:50.165518+0000","last_fullsized":"2024-01-21T05:05:50.165518+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"1109'78","last_scrub_stamp":"2024-01-21T05:05:50.164075+0000","last_deep_scrub":"1109'78","last_deep_scrub_stamp":"2024-01-21T05:05:50.164075+0000","last_clean_scrub_stamp":"2024-01-21T05:05:50.164075+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:40:51.558576+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1059,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.517262+0000","last_change":"2024-01-21T04:33:48.369523+0000","last_active":"2024-01-21T05:05:44.517262+0000","last_peered":"2024-01-21T05:05:44.517262+0000","last_clean":"2024-01-21T05:05:44.517262+0000","last_became_active":"2024-01-21T04:33:48.364387+0000","last_became_peered":"2024-01-21T04:33:48.364387+0000","last_unstale":"2024-01-21T05:05:44.517262+0000","last_undegraded":"2024-01-21T05:05:44.517262+0000","last_fullsized":"2024-01-21T05:05:44.517262+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:06:07.142172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1047,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:27.060750+0000","last_change":"2024-01-21T04:34:54.460251+0000","last_active":"2024-01-21T05:05:27.060750+0000","last_peered":"2024-01-21T05:05:27.060750+0000","last_clean":"2024-01-21T05:05:27.060750+0000","last_became_active":"2024-01-21T04:34:54.452154+0000","last_became_peered":"2024-01-21T04:34:54.452154+0000","last_unstale":"2024-01-21T05:05:27.060750+0000","last_undegraded":"2024-01-21T05:05:27.060750+0000","last_fullsized":"2024-01-21T05:05:27.060750+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:03:09.023273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1059,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.432579+0000","last_change":"2024-01-21T04:33:47.762316+0000","last_active":"2024-01-21T05:05:44.432579+0000","last_peered":"2024-01-21T05:05:44.432579+0000","last_clean":"2024-01-21T05:05:44.432579+0000","last_became_active":"2024-01-21T04:33:47.734733+0000","last_became_peered":"2024-01-21T04:33:47.734733+0000","last_unstale":"2024-01-21T05:05:44.432579+0000","last_undegraded":"2024-01-21T05:05:44.432579+0000","last_fullsized":"2024-01-21T05:05:44.432579+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:16:54.133000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1051,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.600029+0000","last_change":"2024-01-21T04:35:05.297742+0000","last_active":"2024-01-21T05:05:44.600029+0000","last_peered":"2024-01-21T05:05:44.600029+0000","last_clean":"2024-01-21T05:05:44.600029+0000","last_became_active":"2024-01-21T04:35:05.290888+0000","last_became_peered":"2024-01-21T04:35:05.290888+0000","last_unstale":"2024-01-21T05:05:44.600029+0000","last_undegraded":"2024-01-21T05:05:44.600029+0000","last_fullsized":"2024-01-21T05:05:44.600029+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:51:16.216402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1054,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.951406+0000","last_change":"2024-01-21T04:33:48.697481+0000","last_active":"2024-01-21T05:05:26.951406+0000","last_peered":"2024-01-21T05:05:26.951406+0000","last_clean":"2024-01-21T05:05:26.951406+0000","last_became_active":"2024-01-21T04:33:48.670864+0000","last_became_peered":"2024-01-21T04:33:48.670864+0000","last_unstale":"2024-01-21T05:05:26.951406+0000","last_undegraded":"2024-01-21T05:05:26.951406+0000","last_fullsized":"2024-01-21T05:05:26.951406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:00:07.393617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1047,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:27.017847+0000","last_change":"2024-01-21T04:35:07.348347+0000","last_active":"2024-01-21T05:05:27.017847+0000","last_peered":"2024-01-21T05:05:27.017847+0000","last_clean":"2024-01-21T05:05:27.017847+0000","last_became_active":"2024-01-21T04:35:07.340277+0000","last_became_peered":"2024-01-21T04:35:07.340277+0000","last_unstale":"2024-01-21T05:05:27.017847+0000","last_undegraded":"2024-01-21T05:05:27.017847+0000","last_fullsized":"2024-01-21T05:05:27.017847+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:33:05.933710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1061,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.505693+0000","last_change":"2024-01-21T04:37:13.894971+0000","last_active":"2024-01-21T05:05:44.505693+0000","last_peered":"2024-01-21T05:05:44.505693+0000","last_clean":"2024-01-21T05:05:44.505693+0000","last_became_active":"2024-01-21T04:33:48.015845+0000","last_became_peered":"2024-01-21T04:33:48.015845+0000","last_unstale":"2024-01-21T05:05:44.505693+0000","last_undegraded":"2024-01-21T05:05:44.505693+0000","last_fullsized":"2024-01-21T05:05:44.505693+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T04:42:22.571268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1050,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:43.836852+0000","last_change":"2024-01-21T04:35:16.132218+0000","last_active":"2024-01-21T05:05:43.836852+0000","last_peered":"2024-01-21T05:05:43.836852+0000","last_clean":"2024-01-21T05:05:43.836852+0000","last_became_active":"2024-01-21T04:35:16.127034+0000","last_became_peered":"2024-01-21T04:35:16.127034+0000","last_unstale":"2024-01-21T05:05:43.836852+0000","last_undegraded":"2024-01-21T05:05:43.836852+0000","last_fullsized":"2024-01-21T05:05:43.836852+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:29:27.366404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1055,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.493713+0000","last_change":"2024-01-21T04:33:48.068776+0000","last_active":"2024-01-21T05:05:26.493713+0000","last_peered":"2024-01-21T05:05:26.493713+0000","last_clean":"2024-01-21T05:05:26.493713+0000","last_became_active":"2024-01-21T04:33:48.032152+0000","last_became_peered":"2024-01-21T04:33:48.032152+0000","last_unstale":"2024-01-21T05:05:26.493713+0000","last_undegraded":"2024-01-21T05:05:26.493713+0000","last_fullsized":"2024-01-21T05:05:26.493713+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:12:43.256934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1047,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.957819+0000","last_change":"2024-01-21T04:34:55.542633+0000","last_active":"2024-01-21T05:05:26.957819+0000","last_peered":"2024-01-21T05:05:26.957819+0000","last_clean":"2024-01-21T05:05:26.957819+0000","last_became_active":"2024-01-21T04:34:55.527971+0000","last_became_peered":"2024-01-21T04:34:55.527971+0000","last_unstale":"2024-01-21T05:05:26.957819+0000","last_undegraded":"2024-01-21T05:05:26.957819+0000","last_fullsized":"2024-01-21T05:05:26.957819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:19:42.501825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1058,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.550620+0000","last_change":"2024-01-21T04:33:47.840628+0000","last_active":"2024-01-21T05:05:44.550620+0000","last_peered":"2024-01-21T05:05:44.550620+0000","last_clean":"2024-01-21T05:05:44.550620+0000","last_became_active":"2024-01-21T04:33:47.835639+0000","last_became_peered":"2024-01-21T04:33:47.835639+0000","last_unstale":"2024-01-21T05:05:44.550620+0000","last_undegraded":"2024-01-21T05:05:44.550620+0000","last_fullsized":"2024-01-21T05:05:44.550620+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:55:33.768752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1048,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.953043+0000","last_change":"2024-01-21T04:35:26.902591+0000","last_active":"2024-01-21T05:05:26.953043+0000","last_peered":"2024-01-21T05:05:26.953043+0000","last_clean":"2024-01-21T05:05:26.953043+0000","last_became_active":"2024-01-21T04:35:26.889439+0000","last_became_peered":"2024-01-21T04:35:26.889439+0000","last_unstale":"2024-01-21T05:05:26.953043+0000","last_undegraded":"2024-01-21T05:05:26.953043+0000","last_fullsized":"2024-01-21T05:05:26.953043+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:36:16.061320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1054,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:43.860172+0000","last_change":"2024-01-21T04:33:48.112047+0000","last_active":"2024-01-21T05:05:43.860172+0000","last_peered":"2024-01-21T05:05:43.860172+0000","last_clean":"2024-01-21T05:05:43.860172+0000","last_became_active":"2024-01-21T04:33:48.080898+0000","last_became_peered":"2024-01-21T04:33:48.080898+0000","last_unstale":"2024-01-21T05:05:43.860172+0000","last_undegraded":"2024-01-21T05:05:43.860172+0000","last_fullsized":"2024-01-21T05:05:43.860172+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:54:29.756588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1065,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.482067+0000","last_change":"2024-01-21T04:35:36.906825+0000","last_active":"2024-01-21T05:05:26.482067+0000","last_peered":"2024-01-21T05:05:26.482067+0000","last_clean":"2024-01-21T05:05:26.482067+0000","last_became_active":"2024-01-21T04:35:36.901570+0000","last_became_peered":"2024-01-21T04:35:36.901570+0000","last_unstale":"2024-01-21T05:05:26.482067+0000","last_undegraded":"2024-01-21T05:05:26.482067+0000","last_fullsized":"2024-01-21T05:05:26.482067+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:17:42.698152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1057,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.598476+0000","last_change":"2024-01-21T04:33:48.051441+0000","last_active":"2024-01-21T05:05:44.598476+0000","last_peered":"2024-01-21T05:05:44.598476+0000","last_clean":"2024-01-21T05:05:44.598476+0000","last_became_active":"2024-01-21T04:33:48.046404+0000","last_became_peered":"2024-01-21T04:33:48.046404+0000","last_unstale":"2024-01-21T05:05:44.598476+0000","last_undegraded":"2024-01-21T05:05:44.598476+0000","last_fullsized":"2024-01-21T05:05:44.598476+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:33:46.250873+0000","last_clean_scrub_stamp":"2024-01-21T04:33:46.250873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:19:24.446344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1050,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.681021+0000","last_change":"2024-01-21T04:35:19.621072+0000","last_active":"2024-01-21T05:05:44.681021+0000","last_peered":"2024-01-21T05:05:44.681021+0000","last_clean":"2024-01-21T05:05:44.681021+0000","last_became_active":"2024-01-21T04:35:19.611575+0000","last_became_peered":"2024-01-21T04:35:19.611575+0000","last_unstale":"2024-01-21T05:05:44.681021+0000","last_undegraded":"2024-01-21T05:05:44.681021+0000","last_fullsized":"2024-01-21T05:05:44.681021+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1093,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:52.007025+0000","last_change":"2024-01-21T05:05:52.007785+0000","last_active":"2024-01-21T05:05:52.007025+0000","last_peered":"2024-01-21T05:05:52.007025+0000","last_clean":"2024-01-21T05:05:52.007025+0000","last_became_active":"2024-01-21T04:34:53.495212+0000","last_became_peered":"2024-01-21T04:34:53.495212+0000","last_unstale":"2024-01-21T05:05:52.007025+0000","last_undegraded":"2024-01-21T05:05:52.007025+0000","last_fullsized":"2024-01-21T05:05:52.007025+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:52.005738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:52.005738+0000","last_clean_scrub_stamp":"2024-01-21T05:05:52.005738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:59:35.494176+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1046,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.949909+0000","last_change":"2024-01-21T04:34:58.579556+0000","last_active":"2024-01-21T05:05:26.949909+0000","last_peered":"2024-01-21T05:05:26.949909+0000","last_clean":"2024-01-21T05:05:26.949909+0000","last_became_active":"2024-01-21T04:34:58.492357+0000","last_became_peered":"2024-01-21T04:34:58.492357+0000","last_unstale":"2024-01-21T05:05:26.949909+0000","last_undegraded":"2024-01-21T05:05:26.949909+0000","last_fullsized":"2024-01-21T05:05:26.949909+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:55:43.736268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1046,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.954524+0000","last_change":"2024-01-21T04:35:26.240386+0000","last_active":"2024-01-21T05:05:26.954524+0000","last_peered":"2024-01-21T05:05:26.954524+0000","last_clean":"2024-01-21T05:05:26.954524+0000","last_became_active":"2024-01-21T04:35:25.928650+0000","last_became_peered":"2024-01-21T04:35:25.928650+0000","last_unstale":"2024-01-21T05:05:26.954524+0000","last_undegraded":"2024-01-21T05:05:26.954524+0000","last_fullsized":"2024-01-21T05:05:26.954524+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:38:53.872497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1072,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:51.076628+0000","last_change":"2024-01-21T05:05:51.077414+0000","last_active":"2024-01-21T05:05:51.076628+0000","last_peered":"2024-01-21T05:05:51.076628+0000","last_clean":"2024-01-21T05:05:51.076628+0000","last_became_active":"2024-01-21T04:35:00.387495+0000","last_became_peered":"2024-01-21T04:35:00.387495+0000","last_unstale":"2024-01-21T05:05:51.076628+0000","last_undegraded":"2024-01-21T05:05:51.076628+0000","last_fullsized":"2024-01-21T05:05:51.076628+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:51.075137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:51.075137+0000","last_clean_scrub_stamp":"2024-01-21T05:05:51.075137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:10:50.701439+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1085,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.369632+0000","last_change":"2024-01-21T04:35:16.819308+0000","last_active":"2024-01-21T05:05:44.369632+0000","last_peered":"2024-01-21T05:05:44.369632+0000","last_clean":"2024-01-21T05:05:44.369632+0000","last_became_active":"2024-01-21T04:35:16.679256+0000","last_became_peered":"2024-01-21T04:35:16.679256+0000","last_unstale":"2024-01-21T05:05:44.369632+0000","last_undegraded":"2024-01-21T05:05:44.369632+0000","last_fullsized":"2024-01-21T05:05:44.369632+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1072,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:52.081183+0000","last_change":"2024-01-21T05:05:52.081900+0000","last_active":"2024-01-21T05:05:52.081183+0000","last_peered":"2024-01-21T05:05:52.081183+0000","last_clean":"2024-01-21T05:05:52.081183+0000","last_became_active":"2024-01-21T04:35:25.004049+0000","last_became_peered":"2024-01-21T04:35:25.004049+0000","last_unstale":"2024-01-21T05:05:52.081183+0000","last_undegraded":"2024-01-21T05:05:52.081183+0000","last_fullsized":"2024-01-21T05:05:52.081183+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:52.079749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:52.079749+0000","last_clean_scrub_stamp":"2024-01-21T05:05:52.079749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:44:31.431778+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1051,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.684529+0000","last_change":"2024-01-21T04:35:23.577107+0000","last_active":"2024-01-21T05:05:44.684529+0000","last_peered":"2024-01-21T05:05:44.684529+0000","last_clean":"2024-01-21T05:05:44.684529+0000","last_became_active":"2024-01-21T04:35:23.567833+0000","last_became_peered":"2024-01-21T04:35:23.567833+0000","last_unstale":"2024-01-21T05:05:44.684529+0000","last_undegraded":"2024-01-21T05:05:44.684529+0000","last_fullsized":"2024-01-21T05:05:44.684529+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1048,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.550468+0000","last_change":"2024-01-21T04:34:57.160123+0000","last_active":"2024-01-21T05:05:26.550468+0000","last_peered":"2024-01-21T05:05:26.550468+0000","last_clean":"2024-01-21T05:05:26.550468+0000","last_became_active":"2024-01-21T04:34:57.153977+0000","last_became_peered":"2024-01-21T04:34:57.153977+0000","last_unstale":"2024-01-21T05:05:26.550468+0000","last_undegraded":"2024-01-21T05:05:26.550468+0000","last_fullsized":"2024-01-21T05:05:26.550468+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:18:26.237354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1049,"reported_epoch":1108,"state":"active+clean","last_fresh":"2024-01-21T05:05:43.835339+0000","last_change":"2024-01-21T04:35:28.601417+0000","last_active":"2024-01-21T05:05:43.835339+0000","last_peered":"2024-01-21T05:05:43.835339+0000","last_clean":"2024-01-21T05:05:43.835339+0000","last_became_active":"2024-01-21T04:35:28.583632+0000","last_became_peered":"2024-01-21T04:35:28.583632+0000","last_unstale":"2024-01-21T05:05:43.835339+0000","last_undegraded":"2024-01-21T05:05:43.835339+0000","last_fullsized":"2024-01-21T05:05:43.835339+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:35:42.169701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1085,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:44.348323+0000","last_change":"2024-01-21T04:35:16.742449+0000","last_active":"2024-01-21T05:05:44.348323+0000","last_peered":"2024-01-21T05:05:44.348323+0000","last_clean":"2024-01-21T05:05:44.348323+0000","last_became_active":"2024-01-21T04:35:16.734793+0000","last_became_peered":"2024-01-21T04:35:16.734793+0000","last_unstale":"2024-01-21T05:05:44.348323+0000","last_undegraded":"2024-01-21T05:05:44.348323+0000","last_fullsized":"2024-01-21T05:05:44.348323+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1065,"reported_epoch":1107,"state":"active+clean","last_fresh":"2024-01-21T05:05:26.517921+0000","last_change":"2024-01-21T04:35:14.711183+0000","last_active":"2024-01-21T05:05:26.517921+0000","last_peered":"2024-01-21T05:05:26.517921+0000","last_clean":"2024-01-21T05:05:26.517921+0000","last_became_active":"2024-01-21T04:35:14.685816+0000","last_became_peered":"2024-01-21T04:35:14.685816+0000","last_unstale":"2024-01-21T05:05:26.517921+0000","last_undegraded":"2024-01-21T05:05:26.517921+0000","last_fullsized":"2024-01-21T05:05:26.517921+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T04:34:45.585219+0000","last_clean_scrub_stamp":"2024-01-21T04:34:45.585219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:04:28.368206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":21,"seq":90194313605,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74504,"kb_used_data":10904,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297336,"statfs":{"total":96636764160,"available":96560472064,"internally_reserved":0,"allocated":11165696,"data_stored":6053593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":76,"apply_latency_ms":76,"commit_latency_ns":76000000,"apply_latency_ns":76000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:02 2024","interfaces":[{"interface":"back","average":{"1min":16.902000000000001,"5min":11.343,"15min":17.483000000000001},"min":{"1min":6.3620000000000001,"5min":5.4320000000000004,"15min":4.5259999999999998},"max":{"1min":130.69499999999999,"5min":130.69499999999999,"15min":305.86900000000003},"last":7.2190000000000003},{"interface":"front","average":{"1min":19.975999999999999,"5min":14.539,"15min":20.905000000000001},"min":{"1min":10.542999999999999,"5min":8.0370000000000008,"15min":4.6710000000000003},"max":{"1min":121.011,"5min":121.011,"15min":281.79000000000002},"last":47.389000000000003}]},{"osd":1,"last update":"Sun Jan 21 05:05:28 2024","interfaces":[{"interface":"back","average":{"1min":30.696999999999999,"5min":16.850999999999999,"15min":19.510000000000002},"min":{"1min":9.0579999999999998,"5min":7.6550000000000002,"15min":6.1280000000000001},"max":{"1min":151.977,"5min":151.977,"15min":249.82599999999999},"last":51.350999999999999},{"interface":"front","average":{"1min":21.411999999999999,"5min":10.917,"15min":14.314},"min":{"1min":5.5709999999999997,"5min":4.8860000000000001,"15min":4.4429999999999996},"max":{"1min":133.238,"5min":133.238,"15min":185.57599999999999},"last":8.5890000000000004}]},{"osd":2,"last update":"Sun Jan 21 05:05:16 2024","interfaces":[{"interface":"back","average":{"1min":18.077000000000002,"5min":11.068,"15min":16.349},"min":{"1min":6.9770000000000003,"5min":6.4000000000000004,"15min":6.4000000000000004},"max":{"1min":102.258,"5min":102.258,"15min":260.14999999999998},"last":46.512999999999998},{"interface":"front","average":{"1min":24.271000000000001,"5min":15.379,"15min":22.163},"min":{"1min":9.6739999999999995,"5min":7.8390000000000004,"15min":7.8390000000000004},"max":{"1min":108.218,"5min":108.218,"15min":261.67200000000003},"last":50.692}]},{"osd":4,"last update":"Sun Jan 21 05:05:07 2024","interfaces":[{"interface":"back","average":{"1min":21.986999999999998,"5min":12.903,"15min":16.385999999999999},"min":{"1min":6.5780000000000003,"5min":6.1299999999999999,"15min":6.0830000000000002},"max":{"1min":173.92099999999999,"5min":173.92099999999999,"15min":179.381},"last":30.805},{"interface":"front","average":{"1min":22.701000000000001,"5min":15.462,"15min":18.68},"min":{"1min":11.153,"5min":7.0839999999999996,"15min":6.766},"max":{"1min":120.02800000000001,"5min":120.02800000000001,"15min":209.67099999999999},"last":29.536000000000001}]},{"osd":5,"last update":"Sun Jan 21 05:04:58 2024","interfaces":[{"interface":"back","average":{"1min":15.048,"5min":15.337,"15min":22.681999999999999},"min":{"1min":11.866,"5min":7.8719999999999999,"15min":6.9800000000000004},"max":{"1min":17.161999999999999,"5min":30.702000000000002,"15min":318.983},"last":22.001000000000001},{"interface":"front","average":{"1min":12.109,"5min":12.140000000000001,"15min":21.193999999999999},"min":{"1min":8.077,"5min":6.7939999999999996,"15min":6.7939999999999996},"max":{"1min":18.620999999999999,"5min":21.742000000000001,"15min":341.21699999999998},"last":18.527000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:05:03 2024","interfaces":[{"interface":"back","average":{"1min":23.292000000000002,"5min":17.651,"15min":26.388999999999999},"min":{"1min":13.853999999999999,"5min":10.154999999999999,"15min":7.1859999999999999},"max":{"1min":110.64100000000001,"5min":110.64100000000001,"15min":329.39100000000002},"last":56.170999999999999},{"interface":"front","average":{"1min":24.414000000000001,"5min":19.693999999999999,"15min":27.457000000000001},"min":{"1min":13.991,"5min":11.544,"15min":11.026},"max":{"1min":102.995,"5min":102.995,"15min":330.54899999999998},"last":56.850000000000001}]},{"osd":7,"last update":"Sun Jan 21 05:05:28 2024","interfaces":[{"interface":"back","average":{"1min":40.505000000000003,"5min":19.132000000000001,"15min":21.704000000000001},"min":{"1min":9.1530000000000005,"5min":7.8639999999999999,"15min":7.7240000000000002},"max":{"1min":233.38499999999999,"5min":233.38499999999999,"15min":233.38499999999999},"last":10.33},{"interface":"front","average":{"1min":41.484999999999999,"5min":21.462,"15min":23.84},"min":{"1min":9.9809999999999999,"5min":9.9120000000000008,"15min":9.9120000000000008},"max":{"1min":264.34800000000001,"5min":264.34800000000001,"15min":264.34800000000001},"last":31.731999999999999}]}]},{"osd":6,"up_from":20,"seq":85899346310,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74260,"kb_used_data":11148,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297580,"statfs":{"total":96636764160,"available":96560721920,"internally_reserved":0,"allocated":11415552,"data_stored":6213902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":21.414000000000001,"5min":10.807,"15min":18.202999999999999},"min":{"1min":5.2009999999999996,"5min":4.109,"15min":4.0830000000000002},"max":{"1min":170.822,"5min":170.822,"15min":293.416},"last":7.2690000000000001},{"interface":"front","average":{"1min":23.597000000000001,"5min":11.395,"15min":18.113},"min":{"1min":5.6059999999999999,"5min":4.3849999999999998,"15min":4.2220000000000004},"max":{"1min":170.15700000000001,"5min":170.15700000000001,"15min":258.678},"last":14.627000000000001}]},{"osd":1,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":22.026,"5min":13.266,"15min":17.341000000000001},"min":{"1min":7.5270000000000001,"5min":6.2839999999999998,"15min":6.0650000000000004},"max":{"1min":75.444999999999993,"5min":75.444999999999993,"15min":267.21899999999999},"last":10.48},{"interface":"front","average":{"1min":31.890999999999998,"5min":17.786000000000001,"15min":22.844999999999999},"min":{"1min":10.311999999999999,"5min":8.6660000000000004,"15min":7.3339999999999996},"max":{"1min":151.76499999999999,"5min":151.76499999999999,"15min":271.875},"last":14.307}]},{"osd":2,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":23.411999999999999,"5min":13.505000000000001,"15min":20.018000000000001},"min":{"1min":9.3260000000000005,"5min":6.0039999999999996,"15min":6.0039999999999996},"max":{"1min":63.945999999999998,"5min":63.945999999999998,"15min":274.54000000000002},"last":15.99},{"interface":"front","average":{"1min":24.706,"5min":16.004000000000001,"15min":18.666},"min":{"1min":11.185,"5min":7.5869999999999997,"15min":5.9420000000000002},"max":{"1min":71.260999999999996,"5min":71.260999999999996,"15min":163.05199999999999},"last":11.185}]},{"osd":3,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":32.323,"5min":16.699000000000002,"15min":18.744},"min":{"1min":8.1890000000000001,"5min":6.7919999999999998,"15min":6.7549999999999999},"max":{"1min":162.52799999999999,"5min":162.52799999999999,"15min":162.52799999999999},"last":16.771000000000001},{"interface":"front","average":{"1min":26.635999999999999,"5min":17.824000000000002,"15min":22.774999999999999},"min":{"1min":11.757,"5min":10.016,"15min":8.2469999999999999},"max":{"1min":72.900999999999996,"5min":72.900999999999996,"15min":275.81},"last":12.680999999999999}]},{"osd":4,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":30.867999999999999,"5min":19.631,"15min":25.166},"min":{"1min":12.084,"5min":11.298,"15min":9.625},"max":{"1min":128.995,"5min":128.995,"15min":280.101},"last":12.084},{"interface":"front","average":{"1min":32.468000000000004,"5min":21.239000000000001,"15min":22.986000000000001},"min":{"1min":14.935,"5min":11.999000000000001,"15min":6.4119999999999999},"max":{"1min":127.55,"5min":127.55,"15min":273.45600000000002},"last":17.544}]},{"osd":5,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":23.670999999999999,"5min":13.929,"15min":21.393999999999998},"min":{"1min":6.8079999999999998,"5min":6.1699999999999999,"15min":6.1699999999999999},"max":{"1min":98.465999999999994,"5min":98.465999999999994,"15min":275.233},"last":8.4670000000000005},{"interface":"front","average":{"1min":26.530000000000001,"5min":16.762,"15min":22.911999999999999},"min":{"1min":10.563000000000001,"5min":7.9710000000000001,"15min":7.0179999999999998},"max":{"1min":66.147000000000006,"5min":66.147000000000006,"15min":288.76499999999999},"last":15.315}]},{"osd":7,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":24.048999999999999,"5min":14.968999999999999,"15min":22.209},"min":{"1min":8.3249999999999993,"5min":7.5220000000000002,"15min":7.5220000000000002},"max":{"1min":73.034000000000006,"5min":73.034000000000006,"15min":366.68000000000001},"last":11.487},{"interface":"front","average":{"1min":26.388000000000002,"5min":17.59,"15min":23.196999999999999},"min":{"1min":10.593,"5min":9.6989999999999998,"15min":9.6790000000000003},"max":{"1min":74.290999999999997,"5min":74.290999999999997,"15min":281.78500000000003},"last":18.289999999999999}]}]},{"osd":4,"up_from":20,"seq":85899346310,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74504,"kb_used_data":10904,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297336,"statfs":{"total":96636764160,"available":96560472064,"internally_reserved":0,"allocated":11165696,"data_stored":6053593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":17.173999999999999,"5min":11.048999999999999,"15min":16.907},"min":{"1min":5.7629999999999999,"5min":5.7629999999999999,"15min":4.2409999999999997},"max":{"1min":154.04499999999999,"5min":154.04499999999999,"15min":294.92099999999999},"last":7.8300000000000001},{"interface":"front","average":{"1min":18.936,"5min":14.742000000000001,"15min":19.247},"min":{"1min":8.673,"5min":6.4379999999999997,"15min":4.6699999999999999},"max":{"1min":99.890000000000001,"5min":99.890000000000001,"15min":292.43700000000001},"last":9.2799999999999994}]},{"osd":1,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":10.364000000000001,"5min":9.9369999999999994,"15min":15.395},"min":{"1min":6.0350000000000001,"5min":5.1669999999999998,"15min":4.1470000000000002},"max":{"1min":60.579999999999998,"5min":60.579999999999998,"15min":221.84999999999999},"last":6.343},{"interface":"front","average":{"1min":12.541,"5min":13.403,"15min":17.917000000000002},"min":{"1min":7.2190000000000003,"5min":7.032,"15min":6.3959999999999999},"max":{"1min":22.050000000000001,"5min":36.454999999999998,"15min":299.11599999999999},"last":18.102}]},{"osd":2,"last update":"Sun Jan 21 05:04:57 2024","interfaces":[{"interface":"back","average":{"1min":13.195,"5min":13.805999999999999,"15min":19.687999999999999},"min":{"1min":7.6289999999999996,"5min":7.0209999999999999,"15min":5.7439999999999998},"max":{"1min":23.762,"5min":30.693999999999999,"15min":293.60199999999998},"last":13.007999999999999},{"interface":"front","average":{"1min":13.038,"5min":11.211,"15min":20.07},"min":{"1min":7.532,"5min":6.4829999999999997,"15min":6.4829999999999997},"max":{"1min":20.181999999999999,"5min":34.270000000000003,"15min":303.94799999999998},"last":13.348000000000001}]},{"osd":3,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":29.370000000000001,"5min":16.681000000000001,"15min":22.192},"min":{"1min":6.0940000000000003,"5min":6.0940000000000003,"15min":6.0940000000000003},"max":{"1min":226.172,"5min":226.172,"15min":340.21100000000001},"last":11.833},{"interface":"front","average":{"1min":36.143999999999998,"5min":16.957999999999998,"15min":22.98},"min":{"1min":9.0310000000000006,"5min":6.6529999999999996,"15min":6.5579999999999998},"max":{"1min":211.38300000000001,"5min":211.38300000000001,"15min":302.72500000000002},"last":10.023999999999999}]},{"osd":5,"last update":"Sun Jan 21 05:04:58 2024","interfaces":[{"interface":"back","average":{"1min":23.199999999999999,"5min":15.675000000000001,"15min":23.965},"min":{"1min":8.4740000000000002,"5min":7.2510000000000003,"15min":7.2510000000000003},"max":{"1min":206.16800000000001,"5min":206.16800000000001,"15min":305.17200000000003},"last":16.800999999999998},{"interface":"front","average":{"1min":24.972999999999999,"5min":18.754999999999999,"15min":29.27},"min":{"1min":9.9359999999999999,"5min":9.9359999999999999,"15min":9.9359999999999999},"max":{"1min":192.066,"5min":192.066,"15min":326.19900000000001},"last":18.946999999999999}]},{"osd":6,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":24.899999999999999,"5min":15.957000000000001,"15min":23.010999999999999},"min":{"1min":6.8769999999999998,"5min":6.7409999999999997,"15min":6.7409999999999997},"max":{"1min":204.32900000000001,"5min":204.32900000000001,"15min":248.09800000000001},"last":13.964},{"interface":"front","average":{"1min":27.047000000000001,"5min":18.641999999999999,"15min":26.821000000000002},"min":{"1min":7.7800000000000002,"5min":7.7800000000000002,"15min":7.7800000000000002},"max":{"1min":205.94800000000001,"5min":205.94800000000001,"15min":333.46899999999999},"last":15.409000000000001}]},{"osd":7,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":28.289999999999999,"5min":17.347000000000001,"15min":20.901},"min":{"1min":9.5459999999999994,"5min":7.3970000000000002,"15min":6.5430000000000001},"max":{"1min":260.63499999999999,"5min":260.63499999999999,"15min":312.41199999999998},"last":12.377000000000001},{"interface":"front","average":{"1min":29.556000000000001,"5min":19.172999999999998,"15min":25.035},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":236.80799999999999,"5min":236.80799999999999,"15min":247.44499999999999},"last":16.077999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509830,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74500,"kb_used_data":11144,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297340,"statfs":{"total":96636764160,"available":96560476160,"internally_reserved":0,"allocated":11411456,"data_stored":6213897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:21 2024","interfaces":[{"interface":"back","average":{"1min":26.605,"5min":13.619999999999999,"15min":20.289000000000001},"min":{"1min":4.9279999999999999,"5min":4.46,"15min":3.847},"max":{"1min":244.893,"5min":244.893,"15min":272.279},"last":14.045999999999999},{"interface":"front","average":{"1min":27.978000000000002,"5min":12.045999999999999,"15min":19.513000000000002},"min":{"1min":4.516,"5min":4.0579999999999998,"15min":4.0579999999999998},"max":{"1min":224.22800000000001,"5min":224.22800000000001,"15min":356.99799999999999},"last":6.4960000000000004}]},{"osd":1,"last update":"Sun Jan 21 05:04:54 2024","interfaces":[{"interface":"back","average":{"1min":8.9190000000000005,"5min":8.7639999999999993,"15min":15.289999999999999},"min":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7759999999999998},"max":{"1min":19.866,"5min":19.866,"15min":276.94099999999997},"last":5.7030000000000003},{"interface":"front","average":{"1min":13.728,"5min":13.041,"15min":20.687999999999999},"min":{"1min":7.8600000000000003,"5min":7.4729999999999999,"15min":6.383},"max":{"1min":23.332000000000001,"5min":23.332000000000001,"15min":232.74299999999999},"last":15.250999999999999}]},{"osd":2,"last update":"Sun Jan 21 05:05:21 2024","interfaces":[{"interface":"back","average":{"1min":32.673999999999999,"5min":17.591000000000001,"15min":25.193000000000001},"min":{"1min":8.125,"5min":7.5919999999999996,"15min":7.5919999999999996},"max":{"1min":259.29899999999998,"5min":259.29899999999998,"15min":284.34100000000001},"last":16.09},{"interface":"front","average":{"1min":35.651000000000003,"5min":20.353000000000002,"15min":24.027999999999999},"min":{"1min":14.430999999999999,"5min":11.855,"15min":6.0229999999999997},"max":{"1min":260.87799999999999,"5min":260.87799999999999,"15min":260.87799999999999},"last":17.731999999999999}]},{"osd":3,"last update":"Sun Jan 21 05:05:37 2024","interfaces":[{"interface":"back","average":{"1min":41.347000000000001,"5min":21.199000000000002,"15min":26.056000000000001},"min":{"1min":8.8219999999999992,"5min":8.7919999999999998,"15min":7.8109999999999999},"max":{"1min":263.053,"5min":263.053,"15min":276.61399999999998},"last":16.881},{"interface":"front","average":{"1min":44.539999999999999,"5min":23.657,"15min":29.605},"min":{"1min":14.435,"5min":13.215,"15min":10.497},"max":{"1min":265.154,"5min":265.154,"15min":277.28100000000001},"last":18.526}]},{"osd":4,"last update":"Sun Jan 21 05:05:37 2024","interfaces":[{"interface":"back","average":{"1min":50.813000000000002,"5min":18.835000000000001,"15min":17.899000000000001},"min":{"1min":7.5730000000000004,"5min":5.8620000000000001,"15min":5.8620000000000001},"max":{"1min":316.40699999999998,"5min":316.40699999999998,"15min":316.40699999999998},"last":9.1099999999999994},{"interface":"front","average":{"1min":45.965000000000003,"5min":23.863,"15min":25.629000000000001},"min":{"1min":15.303000000000001,"5min":12.269,"15min":9.1609999999999996},"max":{"1min":256.67099999999999,"5min":256.67099999999999,"15min":263.23099999999999},"last":19.326000000000001}]},{"osd":5,"last update":"Sun Jan 21 05:05:22 2024","interfaces":[{"interface":"back","average":{"1min":38.140999999999998,"5min":16.295000000000002,"15min":21.081},"min":{"1min":8.4489999999999998,"5min":6.6459999999999999,"15min":6.6429999999999998},"max":{"1min":223.56999999999999,"5min":223.56999999999999,"15min":223.56999999999999},"last":11.119},{"interface":"front","average":{"1min":40.212000000000003,"5min":19.149000000000001,"15min":23.727},"min":{"1min":8.6560000000000006,"5min":8.6240000000000006,"15min":8.5429999999999993},"max":{"1min":224.81100000000001,"5min":224.81100000000001,"15min":224.81100000000001},"last":11.837}]},{"osd":6,"last update":"Sun Jan 21 05:04:54 2024","interfaces":[{"interface":"back","average":{"1min":13.615,"5min":13.750999999999999,"15min":26.07},"min":{"1min":8.2759999999999998,"5min":7.2210000000000001,"15min":7.2210000000000001},"max":{"1min":31.048999999999999,"5min":31.048999999999999,"15min":371.17599999999999},"last":13.304},{"interface":"front","average":{"1min":13.039999999999999,"5min":13.173,"15min":19.331},"min":{"1min":7.931,"5min":7.5679999999999996,"15min":7.1020000000000003},"max":{"1min":22.701000000000001,"5min":22.701000000000001,"15min":179.63900000000001},"last":10.398}]}]},{"osd":0,"up_from":15,"seq":64424509826,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78420,"kb_used_data":10904,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94293420,"statfs":{"total":96636764160,"available":96556462080,"internally_reserved":0,"allocated":11165696,"data_stored":6053593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":56.753,"5min":18.943000000000001,"15min":22.891999999999999},"min":{"1min":5.2919999999999998,"5min":4.8209999999999997,"15min":4.5629999999999997},"max":{"1min":310.80200000000002,"5min":310.80200000000002,"15min":310.80200000000002},"last":7.3650000000000002},{"interface":"front","average":{"1min":46.957999999999998,"5min":16.795999999999999,"15min":20.858000000000001},"min":{"1min":7.165,"5min":4.9880000000000004,"15min":4.9880000000000004},"max":{"1min":303.25200000000001,"5min":303.25200000000001,"15min":303.25200000000001},"last":8.9909999999999997}]},{"osd":2,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":40.720999999999997,"5min":16.710999999999999,"15min":23.585000000000001},"min":{"1min":6.7960000000000003,"5min":6.2800000000000002,"15min":5.5469999999999997},"max":{"1min":275.13299999999998,"5min":275.13299999999998,"15min":404.36700000000002},"last":8.2059999999999995},{"interface":"front","average":{"1min":47.176000000000002,"5min":21.809999999999999,"15min":30.907},"min":{"1min":9.7240000000000002,"5min":9.4169999999999998,"15min":8.8650000000000002},"max":{"1min":280.69900000000001,"5min":280.69900000000001,"15min":407.68000000000001},"last":9.7240000000000002}]},{"osd":3,"last update":"Sun Jan 21 05:05:06 2024","interfaces":[{"interface":"back","average":{"1min":26.777000000000001,"5min":17.292000000000002,"15min":25.620999999999999},"min":{"1min":9.8919999999999995,"5min":6.266,"15min":6.266},"max":{"1min":212.34200000000001,"5min":212.34200000000001,"15min":253.25399999999999},"last":16.504000000000001},{"interface":"front","average":{"1min":13.295999999999999,"5min":10.103999999999999,"15min":17.751999999999999},"min":{"1min":5.7930000000000001,"5min":5.7930000000000001,"15min":5.7930000000000001},"max":{"1min":63.148000000000003,"5min":63.148000000000003,"15min":315.92399999999998},"last":13.800000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":27.547000000000001,"5min":14.999000000000001,"15min":20.422999999999998},"min":{"1min":6.7510000000000003,"5min":6.7510000000000003,"15min":6.2089999999999996},"max":{"1min":95.712999999999994,"5min":95.712999999999994,"15min":283.37599999999998},"last":12.462},{"interface":"front","average":{"1min":56.161999999999999,"5min":24.68,"15min":30.161000000000001},"min":{"1min":12.042999999999999,"5min":9.7360000000000007,"15min":9.7360000000000007},"max":{"1min":336.72899999999998,"5min":336.72899999999998,"15min":344.37200000000001},"last":15.753}]},{"osd":5,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":32.240000000000002,"5min":17.824000000000002,"15min":26.983000000000001},"min":{"1min":9.1850000000000005,"5min":7.4290000000000003,"15min":6.6029999999999998},"max":{"1min":211.78800000000001,"5min":211.78800000000001,"15min":354.04300000000001},"last":17.866},{"interface":"front","average":{"1min":27.254999999999999,"5min":15.403,"15min":23.954000000000001},"min":{"1min":8.7759999999999998,"5min":7.7089999999999996,"15min":6.3159999999999998},"max":{"1min":192.12899999999999,"5min":192.12899999999999,"15min":254.476},"last":11.766}]},{"osd":6,"last update":"Sun Jan 21 05:05:02 2024","interfaces":[{"interface":"back","average":{"1min":25.177,"5min":16.742000000000001,"15min":24.870000000000001},"min":{"1min":10.148,"5min":8.1799999999999997,"15min":6.5659999999999998},"max":{"1min":226.14599999999999,"5min":226.14599999999999,"15min":288.29599999999999},"last":13.163},{"interface":"front","average":{"1min":27.905999999999999,"5min":19.323,"15min":29.056000000000001},"min":{"1min":11.382,"5min":10.058,"15min":8.8550000000000004},"max":{"1min":236.083,"5min":236.083,"15min":293.56400000000002},"last":18.491}]},{"osd":7,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":28.417999999999999,"5min":15.923999999999999,"15min":23.308},"min":{"1min":9.2400000000000002,"5min":8.0700000000000003,"15min":7.6509999999999998},"max":{"1min":113.319,"5min":113.319,"15min":329.11200000000002},"last":19.861999999999998},{"interface":"front","average":{"1min":33.444000000000003,"5min":19.013000000000002,"15min":28.425999999999998},"min":{"1min":10.021000000000001,"5min":10.021000000000001,"15min":9.3699999999999992},"max":{"1min":115.97,"5min":115.97,"15min":337.24000000000001},"last":20.728999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509829,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82336,"kb_used_data":10904,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289504,"statfs":{"total":96636764160,"available":96552452096,"internally_reserved":0,"allocated":11165696,"data_stored":6053593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":14.563000000000001,"5min":11.273999999999999,"15min":15.429},"min":{"1min":4.4800000000000004,"5min":4.4800000000000004,"15min":4.343},"max":{"1min":68.804000000000002,"5min":68.804000000000002,"15min":188.334},"last":7.6040000000000001},{"interface":"front","average":{"1min":10.598000000000001,"5min":9.1699999999999999,"15min":14.536},"min":{"1min":5.8540000000000001,"5min":5.0380000000000003,"15min":4.2919999999999998},"max":{"1min":26.100000000000001,"5min":26.100000000000001,"15min":153.12299999999999},"last":11.750999999999999}]},{"osd":1,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":13.564,"5min":10.316000000000001,"15min":16.317},"min":{"1min":5.6440000000000001,"5min":5.5419999999999998,"15min":5.5419999999999998},"max":{"1min":37.771999999999998,"5min":37.771999999999998,"15min":318.404},"last":16.763999999999999},{"interface":"front","average":{"1min":19.129000000000001,"5min":14.199,"15min":21.916},"min":{"1min":10.805999999999999,"5min":7.1699999999999999,"15min":7.1699999999999999},"max":{"1min":58.195,"5min":58.195,"15min":333.91000000000003},"last":14.74}]},{"osd":3,"last update":"Sun Jan 21 05:05:24 2024","interfaces":[{"interface":"back","average":{"1min":17.065000000000001,"5min":12.718,"15min":18.966999999999999},"min":{"1min":7.1639999999999997,"5min":6.9340000000000002,"15min":5.7240000000000002},"max":{"1min":48.807000000000002,"5min":48.807000000000002,"15min":314.58600000000001},"last":18.262},{"interface":"front","average":{"1min":19.007000000000001,"5min":15.404999999999999,"15min":22.751000000000001},"min":{"1min":9.9469999999999992,"5min":9.7530000000000001,"15min":8.5250000000000004},"max":{"1min":71.736999999999995,"5min":71.736999999999995,"15min":335.71699999999998},"last":16.317}]},{"osd":4,"last update":"Sun Jan 21 05:05:43 2024","interfaces":[{"interface":"back","average":{"1min":19.684000000000001,"5min":14.772,"15min":20.323},"min":{"1min":9.1600000000000001,"5min":8.25,"15min":6.0999999999999996},"max":{"1min":54.728999999999999,"5min":54.728999999999999,"15min":313.17200000000003},"last":13.957000000000001},{"interface":"front","average":{"1min":21.738,"5min":17.106000000000002,"15min":22.914999999999999},"min":{"1min":12.092000000000001,"5min":9.593,"15min":7.6070000000000002},"max":{"1min":75.061999999999998,"5min":75.061999999999998,"15min":331.18200000000002},"last":17.652000000000001}]},{"osd":5,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":20.053999999999998,"5min":12.343999999999999,"15min":19.91},"min":{"1min":6.8529999999999998,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":166.327,"5min":166.327,"15min":270.91399999999999},"last":9.2029999999999994},{"interface":"front","average":{"1min":24.475999999999999,"5min":17.946000000000002,"15min":24.292999999999999},"min":{"1min":13.491,"5min":10.361000000000001,"15min":8.8279999999999994},"max":{"1min":130.548,"5min":130.548,"15min":259.60399999999998},"last":18.933}]},{"osd":6,"last update":"Sun Jan 21 05:04:51 2024","interfaces":[{"interface":"back","average":{"1min":17.291,"5min":17.452999999999999,"15min":26.175000000000001},"min":{"1min":12.855,"5min":11.922000000000001,"15min":7.9800000000000004},"max":{"1min":20.937000000000001,"5min":31.047000000000001,"15min":321.79500000000002},"last":20.131},{"interface":"front","average":{"1min":12.227,"5min":12.388,"15min":21.454999999999998},"min":{"1min":8.5169999999999995,"5min":7.9699999999999998,"15min":7.7229999999999999},"max":{"1min":20.093,"5min":28.638999999999999,"15min":282.54599999999999},"last":10.646000000000001}]},{"osd":7,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":24.565000000000001,"5min":19.756,"15min":26.001000000000001},"min":{"1min":16.887,"5min":13.028,"15min":8.8460000000000001},"max":{"1min":67.106999999999999,"5min":67.106999999999999,"15min":312.435},"last":21.044},{"interface":"front","average":{"1min":20.120000000000001,"5min":15.542,"15min":23.835000000000001},"min":{"1min":8.5540000000000003,"5min":8.2370000000000001,"15min":8.2370000000000001},"max":{"1min":97.379000000000005,"5min":97.379000000000005,"15min":256.11900000000003},"last":12.972}]}]},{"osd":1,"up_from":15,"seq":64424509827,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74652,"kb_used_data":11540,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297188,"statfs":{"total":96636764160,"available":96560320512,"internally_reserved":0,"allocated":11816960,"data_stored":6611742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":11.103999999999999,"5min":9.2059999999999995,"15min":15.691000000000001},"min":{"1min":4.1760000000000002,"5min":4.1760000000000002,"15min":4.1760000000000002},"max":{"1min":63.862000000000002,"5min":63.862000000000002,"15min":209.96299999999999},"last":9.9480000000000004},{"interface":"front","average":{"1min":12.346,"5min":9.9589999999999996,"15min":16.739000000000001},"min":{"1min":5.7009999999999996,"5min":4.9560000000000004,"15min":4.7549999999999999},"max":{"1min":51.240000000000002,"5min":51.240000000000002,"15min":244.054},"last":12.614000000000001}]},{"osd":2,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":25.413,"5min":13.148999999999999,"15min":17.792999999999999},"min":{"1min":8.1020000000000003,"5min":6.202,"15min":4.8819999999999997},"max":{"1min":218.80500000000001,"5min":218.80500000000001,"15min":221.54300000000001},"last":16.039000000000001},{"interface":"front","average":{"1min":24.704999999999998,"5min":15.188000000000001,"15min":20.256},"min":{"1min":10.317,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":144.12899999999999,"5min":144.12899999999999,"15min":219.34899999999999},"last":20.984000000000002}]},{"osd":3,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":14.996,"5min":13.903,"15min":21.224},"min":{"1min":7.3200000000000003,"5min":6.569,"15min":6.569},"max":{"1min":40.685000000000002,"5min":40.685000000000002,"15min":163.58799999999999},"last":15.41},{"interface":"front","average":{"1min":14.272,"5min":12.506,"15min":19.687999999999999},"min":{"1min":7.9630000000000001,"5min":7.0549999999999997,"15min":6.4809999999999999},"max":{"1min":55.024999999999999,"5min":55.024999999999999,"15min":244.178},"last":11.244}]},{"osd":4,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":20.52,"5min":16.398,"15min":21.395},"min":{"1min":11.083,"5min":7.6369999999999996,"15min":7.6369999999999996},"max":{"1min":89.608000000000004,"5min":89.608000000000004,"15min":241.94800000000001},"last":20.125},{"interface":"front","average":{"1min":16.532,"5min":11.914,"15min":18.238},"min":{"1min":6.6500000000000004,"5min":6.6500000000000004,"15min":6.6500000000000004},"max":{"1min":112.226,"5min":112.226,"15min":225.60499999999999},"last":13.615}]},{"osd":5,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":26.631,"5min":16.741,"15min":22.994},"min":{"1min":9.1850000000000005,"5min":9.1850000000000005,"15min":6.8970000000000002},"max":{"1min":156.554,"5min":156.554,"15min":313.36599999999999},"last":17.933},{"interface":"front","average":{"1min":26.550000000000001,"5min":16.300999999999998,"15min":23.606000000000002},"min":{"1min":8.9369999999999994,"5min":8.9369999999999994,"15min":5.851},"max":{"1min":138.114,"5min":138.114,"15min":298.49599999999998},"last":14.798999999999999}]},{"osd":6,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":21.718,"5min":16.562999999999999,"15min":24.361999999999998},"min":{"1min":11.117000000000001,"5min":8.8949999999999996,"15min":8.8949999999999996},"max":{"1min":69.263999999999996,"5min":69.263999999999996,"15min":298.51999999999998},"last":16.692},{"interface":"front","average":{"1min":36.866,"5min":21.071999999999999,"15min":25.277000000000001},"min":{"1min":13.145,"5min":11.101000000000001,"15min":7.4880000000000004},"max":{"1min":293.34699999999998,"5min":293.34699999999998,"15min":293.34699999999998},"last":17.308}]},{"osd":7,"last update":"Sun Jan 21 05:05:17 2024","interfaces":[{"interface":"back","average":{"1min":21.521000000000001,"5min":16.407,"15min":24.297999999999998},"min":{"1min":11.332000000000001,"5min":9.8070000000000004,"15min":9.3230000000000004},"max":{"1min":107.875,"5min":107.875,"15min":214.24199999999999},"last":14.228},{"interface":"front","average":{"1min":20.009,"5min":17.219999999999999,"15min":24.443000000000001},"min":{"1min":9.5779999999999994,"5min":9.5779999999999994,"15min":9.4120000000000008},"max":{"1min":51.546999999999997,"5min":51.546999999999997,"15min":180.73099999999999},"last":22.074999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575235,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74648,"kb_used_data":11536,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297192,"statfs":{"total":96636764160,"available":96560324608,"internally_reserved":0,"allocated":11812864,"data_stored":6611737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":23.058,"5min":12.195,"15min":18.460000000000001},"min":{"1min":6.226,"5min":5.2300000000000004,"15min":4.601},"max":{"1min":154.13399999999999,"5min":154.13399999999999,"15min":377.50999999999999},"last":14.146000000000001},{"interface":"front","average":{"1min":32.405999999999999,"5min":17.396999999999998,"15min":21.706},"min":{"1min":7.8029999999999999,"5min":5.8559999999999999,"15min":4.4359999999999999},"max":{"1min":193.01400000000001,"5min":193.01400000000001,"15min":224.58600000000001},"last":18.012}]},{"osd":1,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":24.754999999999999,"5min":14.738,"15min":22.327999999999999},"min":{"1min":6.165,"5min":5.6820000000000004,"15min":4.5380000000000003},"max":{"1min":160.279,"5min":160.279,"15min":368.79199999999997},"last":6.6669999999999998},{"interface":"front","average":{"1min":28.187000000000001,"5min":13.547000000000001,"15min":19.158999999999999},"min":{"1min":9.9169999999999998,"5min":4.944,"15min":4.3170000000000002},"max":{"1min":156.38499999999999,"5min":156.38499999999999,"15min":225.21000000000001},"last":15.984999999999999}]},{"osd":2,"last update":"Sun Jan 21 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":28.533000000000001,"5min":14.004,"15min":20.603999999999999},"min":{"1min":7.7069999999999999,"5min":6.6420000000000003,"15min":5.8460000000000001},"max":{"1min":232.78200000000001,"5min":232.78200000000001,"15min":232.78200000000001},"last":8.6899999999999995},{"interface":"front","average":{"1min":35.103000000000002,"5min":15.885999999999999,"15min":22.988},"min":{"1min":7.9740000000000002,"5min":6.4199999999999999,"15min":5.8289999999999997},"max":{"1min":288.49099999999999,"5min":288.49099999999999,"15min":309.24599999999998},"last":10.571999999999999}]},{"osd":3,"last update":"Sun Jan 21 05:05:45 2024","interfaces":[{"interface":"back","average":{"1min":52.302999999999997,"5min":22.747,"15min":27.129000000000001},"min":{"1min":7.7610000000000001,"5min":7.7610000000000001,"15min":6.7229999999999999},"max":{"1min":271.49000000000001,"5min":271.49000000000001,"15min":375.07799999999997},"last":18.847999999999999},{"interface":"front","average":{"1min":41.609999999999999,"5min":17.966999999999999,"15min":23.789000000000001},"min":{"1min":8.0259999999999998,"5min":6.4589999999999996,"15min":6.4589999999999996},"max":{"1min":257.07600000000002,"5min":257.07600000000002,"15min":276.57100000000003},"last":12.083}]},{"osd":4,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":31.202999999999999,"5min":15.868,"15min":24.077000000000002},"min":{"1min":8.9039999999999999,"5min":6.3360000000000003,"15min":6.3360000000000003},"max":{"1min":184.767,"5min":184.767,"15min":272.274},"last":9.8000000000000007},{"interface":"front","average":{"1min":33.640999999999998,"5min":20.071999999999999,"15min":28.475000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":6.8689999999999998},"max":{"1min":204.47499999999999,"5min":204.47499999999999,"15min":378.404},"last":19.664000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:05:45 2024","interfaces":[{"interface":"back","average":{"1min":48.223999999999997,"5min":19.954999999999998,"15min":25.443999999999999},"min":{"1min":9.4209999999999994,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":305.68299999999999,"5min":305.68299999999999,"15min":305.68299999999999},"last":13.084},{"interface":"front","average":{"1min":62.576000000000001,"5min":25.863,"15min":32.338999999999999},"min":{"1min":12.919,"5min":10.442,"15min":9.6750000000000007},"max":{"1min":333.18700000000001,"5min":333.18700000000001,"15min":333.18700000000001},"last":16.988}]},{"osd":7,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":31.890000000000001,"5min":20.614999999999998,"15min":29.431999999999999},"min":{"1min":10.763,"5min":9.9640000000000004,"15min":8.032},"max":{"1min":202.90700000000001,"5min":202.90700000000001,"15min":486.358},"last":20.440999999999999},{"interface":"front","average":{"1min":36.167000000000002,"5min":20.385999999999999,"15min":27.478999999999999},"min":{"1min":10.587999999999999,"5min":9.8740000000000006,"15min":9.1039999999999992},"max":{"1min":177.25800000000001,"5min":177.25800000000001,"15min":486.62900000000002},"last":14.846}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T05:05:55.588 INFO:tasks.ceph:pgid 12.1c last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.588 INFO:tasks.ceph:pgid 12.1d last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.588 INFO:tasks.ceph:pgid 12.1e last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.588 INFO:tasks.ceph:pgid 12.1f last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.589 INFO:tasks.ceph:pgid 12.0 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.589 INFO:tasks.ceph:pgid 12.1 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.589 INFO:tasks.ceph:pgid 12.3 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.589 INFO:tasks.ceph:pgid 12.4 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.589 INFO:tasks.ceph:pgid 12.6 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.589 INFO:tasks.ceph:pgid 12.7 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.589 INFO:tasks.ceph:pgid 12.d last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.589 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-01-21T04:33:46.250873+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=33, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.589 INFO:tasks.ceph:pgid 12.e last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.590 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-01-21T04:33:46.250873+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=33, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.590 INFO:tasks.ceph:pgid 12.f last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.590 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-01-21T04:33:46.250873+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=33, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.590 INFO:tasks.ceph:pgid 12.c last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.590 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-01-21T04:33:46.250873+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=33, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.590 INFO:tasks.ceph:pgid 12.a last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.590 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-01-21T04:33:46.250873+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=33, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.590 INFO:tasks.ceph:pgid 12.b last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.590 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-01-21T04:33:46.250873+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=33, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.590 INFO:tasks.ceph:pgid 12.8 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.591 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-01-21T04:33:46.250873+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=33, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.591 INFO:tasks.ceph:pgid 12.9 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.591 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-01-21T04:33:46.250873+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=33, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.591 INFO:tasks.ceph:pgid 12.1b last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.591 INFO:tasks.ceph:pgid 12.19 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.591 INFO:tasks.ceph:pgid 12.18 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.591 INFO:tasks.ceph:pgid 12.16 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.591 INFO:tasks.ceph:pgid 12.14 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.592 INFO:tasks.ceph:pgid 12.13 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.592 INFO:tasks.ceph:pgid 12.12 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.592 INFO:tasks.ceph:pgid 12.11 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.592 INFO:tasks.ceph:pgid 12.10 last_scrub_stamp 2024-01-21T04:34:45.585219+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=4, tm_min=34, tm_sec=45, tm_wday=6, tm_yday=21, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=21, tm_hour=5, tm_min=5, tm_sec=46, tm_wday=6, tm_yday=21, tm_isdst=0) 2024-01-21T05:05:55.592 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-21T05:06:04.412 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:32.318 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:04.414 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:32.321 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:04.429 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:32.336 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:04.461 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:32.336 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:04.462 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:32.337 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:04.476 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:32.351 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:04.480 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:32.387 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:04.481 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:32.389 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:04.497 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:32.404 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:09.419 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:37.325 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:09.422 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:37.329 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:09.434 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:37.341 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:09.463 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:37.338 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:09.468 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:37.343 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:09.469 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:37.376 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:09.472 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:37.379 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:09.478 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:37.354 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:09.483 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:37.391 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:15.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-21T05:06:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.696+0000 7faeb9b97640 10 monclient: get_monmap_and_config 2024-01-21T05:06:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.696+0000 7faeb9b97640 10 monclient: build_initial_monmap 2024-01-21T05:06:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.696+0000 7faeb9b97640 10 monclient: monmap: 2024-01-21T05:06:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:06:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:06:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:06:15.697569+0000 2024-01-21T05:06:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:06:15.697569+0000 2024-01-21T05:06:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:06:15.698 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:15.698 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:06:15.698 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:06:15.698 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:06:15.698 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:06:15.698 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:06:15.698 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 1 Processor -- start 2024-01-21T05:06:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 1 -- start start 2024-01-21T05:06:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 10 monclient: init 2024-01-21T05:06:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 10 monclient: _reopen_session rank -1 2024-01-21T05:06:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-01-21T05:06:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faeb40b02b0 0x7faeb413edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 10 monclient(hunting): picked mon.noname-a con 0x7faeb40b02b0 addr v2:172.21.15.69:3300/0 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faeb40ad230 0x7faeb413f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 10 monclient(hunting): picked mon.noname-c con 0x7faeb40ad230 addr v2:172.21.15.144:3300/0 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faeb4145670 addr v1:172.21.15.69:6789/0 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb413f850 con 0x7faeb4145670 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 10 monclient(hunting): start opening mon connection 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb40ae5f0 con 0x7faeb40b02b0 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 10 monclient(hunting): start opening mon connection 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb40ae8d0 con 0x7faeb40ad230 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 10 monclient(hunting): _renew_subs 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.697+0000 7faeb9b97640 10 monclient(hunting): authenticate will time out at 3334.039195s 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.698+0000 7faeb37fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faeb40b02b0 0x7faeb413edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.698+0000 7faeb2ffd640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7faeb40ad230 0x7faeb413f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.698+0000 7faeb37fe640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faeb40b02b0 0x7faeb413edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53074/0 (socket says 172.21.15.69:53074) 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.698+0000 7faeb37fe640 1 -- 172.21.15.69:0/3694779593 learned_addr learned my addr 172.21.15.69:0/3694779593 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:06:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.698+0000 7faeb37fe640 10 monclient(hunting): get_auth_request con 0x7faeb40b02b0 auth_method 0 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.698+0000 7faeb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.698+0000 7faeb37fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.698+0000 7faeb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.699+0000 7faeb2ffd640 10 monclient(hunting): get_auth_request con 0x7faeb40ad230 auth_method 0 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.699+0000 7faeb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.699+0000 7faeb2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.699+0000 7faeb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.699+0000 7faeb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.699+0000 7faeb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.699+0000 7faeb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.699+0000 7faeb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.699+0000 7faeb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:06:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.699+0000 7faeb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:06:15.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.701+0000 7faeb37fe640 10 monclient(hunting): handle_auth_done global_id 17520 payload 274 2024-01-21T05:06:15.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.701+0000 7faeb37fe640 1 -- 172.21.15.69:0/3694779593 >> v2:172.21.15.144:3300/0 conn(0x7faeb40ad230 msgr2=0x7faeb413f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:06:15.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.701+0000 7faeb37fe640 1 --2- 172.21.15.69:0/3694779593 >> v2:172.21.15.144:3300/0 conn(0x7faeb40ad230 0x7faeb413f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.701+0000 7faeb37fe640 1 -- 172.21.15.69:0/3694779593 >> v1:172.21.15.69:6789/0 conn(0x7faeb4145670 legacy=0x7faeb4145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:06:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.701+0000 7faeb37fe640 10 monclient: _finish_hunting 0 2024-01-21T05:06:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.701+0000 7faeb37fe640 1 monclient: found mon.noname-a 2024-01-21T05:06:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.701+0000 7faeb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:06:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.701+0000 7faeb37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.69:3300/0 2024-01-21T05:06:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.701+0000 7faeb37fe640 1 -- 172.21.15.69:0/3694779593 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb40ad750 con 0x7faeb40b02b0 2024-01-21T05:06:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.703+0000 7faeb37fe640 1 --2- 172.21.15.69:0/3694779593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faeb40b02b0 0x7faeb413edd0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7faeb40b0ac0 tx=0x7faea402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbf4ed62aa654af server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 1 -- 172.21.15.69:0/3694779593 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4031870 con 0x7faeb40b02b0 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 10 monclient: dump: 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:06:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 10 monclient: _finish_auth 0 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 10 monclient: _check_auth_tickets 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 1 -- 172.21.15.69:0/3694779593 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea40319d0 con 0x7faeb40b02b0 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 5 monclient: authenticate success, global_id 17520 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 1 -- 172.21.15.69:0/3694779593 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4031d60 con 0x7faeb40b02b0 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb27fc640 10 monclient: dump: 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:06:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:06:15.710 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:15.710 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:15.710 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:06:15.710 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:15.710 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:06:15.710 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:06:15.710 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:06:15.710 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:15.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 10 monclient: get_monmap_and_config success 2024-01-21T05:06:15.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 4 set_mon_vals no callback set 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 10 monclient: shutdown 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 1 -- 172.21.15.69:0/3694779593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faeb40b02b0 msgr2=0x7faeb413edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 1 --2- 172.21.15.69:0/3694779593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faeb40b02b0 0x7faeb413edd0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7faeb40b0ac0 tx=0x7faea402b910 comp rx=0 tx=0).stop 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 1 -- 172.21.15.69:0/3694779593 shutdown_connections 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 1 --2- 172.21.15.69:0/3694779593 >> v2:172.21.15.144:3300/0 conn(0x7faeb40ad230 0x7faeb413f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 1 --2- 172.21.15.69:0/3694779593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faeb40b02b0 0x7faeb413edd0 unknown :-1 s=CLOSED pgs=3509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 1 -- 172.21.15.69:0/3694779593 >> 172.21.15.69:0/3694779593 conn(0x7faeb40b1130 msgr2=0x7faeb40b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 1 -- 172.21.15.69:0/3694779593 shutdown_connections 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.707+0000 7faeb9b97640 1 -- 172.21.15.69:0/3694779593 wait complete. 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.708+0000 7faeb9b97640 10 monclient: build_initial_monmap 2024-01-21T05:06:15.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.708+0000 7faeb9b97640 10 monclient: monmap: 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.708+0000 7faeb9b97640 1 librados: starting msgr at 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.708+0000 7faeb9b97640 1 librados: starting objecter 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.708+0000 7faeb9b97640 1 Processor -- start 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 1 -- start start 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 1 librados: setting wanted keys 2024-01-21T05:06:15.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 1 librados: calling monclient init 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 10 monclient: init 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 10 monclient: _reopen_session rank -1 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 10 monclient: _add_conns ranks=[1,2,0] 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faeb40ad230 0x7faeb4135760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 10 monclient(hunting): picked mon.noname-b con 0x7faeb40ad230 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faeb40b02b0 0x7faeb4144920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 10 monclient(hunting): picked mon.noname-c con 0x7faeb40b02b0 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faeb4145670 0x7faeb4144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 10 monclient(hunting): picked mon.noname-a con 0x7faeb4145670 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 10 monclient(hunting): start opening mon connection 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4143130 con 0x7faeb4145670 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 10 monclient(hunting): start opening mon connection 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb41daea0 con 0x7faeb40ad230 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 10 monclient(hunting): start opening mon connection 2024-01-21T05:06:15.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb41d4c20 con 0x7faeb40b02b0 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 10 monclient(hunting): _renew_subs 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.709+0000 7faeb9b97640 10 monclient(hunting): authenticate will time out at 3334.050745s 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb2ffd640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faeb40b02b0 0x7faeb4144920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faeb4145670 0x7faeb4144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb37fe640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faeb40ad230 0x7faeb4135760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faeb4145670 0x7faeb4144e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53078/0 (socket says 172.21.15.69:53078) 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb3fff640 1 -- 172.21.15.69:0/3787208009 learned_addr learned my addr 172.21.15.69:0/3787208009 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb3fff640 10 monclient(hunting): get_auth_request con 0x7faeb4145670 auth_method 0 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb3fff640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb2ffd640 10 monclient(hunting): get_auth_request con 0x7faeb40b02b0 auth_method 0 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb2ffd640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:06:15.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb37fe640 10 monclient(hunting): get_auth_request con 0x7faeb40ad230 auth_method 0 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb37fe640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.710+0000 7faeb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.711+0000 7faeb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.711+0000 7faeb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.711+0000 7faeb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.711+0000 7faeb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.711+0000 7faeb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.711+0000 7faeb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.711+0000 7faeb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.711+0000 7faeb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:06:15.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.711+0000 7faeb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:06:15.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.713+0000 7faeb37fe640 10 monclient(hunting): handle_auth_done global_id 21637 payload 824 2024-01-21T05:06:15.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.714+0000 7faeb37fe640 1 -- 172.21.15.69:0/3787208009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faeb40b02b0 msgr2=0x7faeb4144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:06:15.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.714+0000 7faeb37fe640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faeb40b02b0 0x7faeb4144920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:15.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.714+0000 7faeb37fe640 1 -- 172.21.15.69:0/3787208009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faeb4145670 msgr2=0x7faeb4144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:06:15.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.714+0000 7faeb37fe640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faeb4145670 0x7faeb4144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:15.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.714+0000 7faeb37fe640 10 monclient: _finish_hunting 0 2024-01-21T05:06:15.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.714+0000 7faeb37fe640 1 monclient: found mon.noname-b 2024-01-21T05:06:15.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.714+0000 7faeb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:06:15.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.714+0000 7faeb37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:06:15.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.714+0000 7faeb37fe640 1 -- 172.21.15.69:0/3787208009 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea40094d0 con 0x7faeb40ad230 2024-01-21T05:06:15.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.716+0000 7faeb37fe640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faeb40ad230 0x7faeb4135760 secure :-1 s=READY pgs=5243 cs=0 l=1 rev1=1 crypto rx=0x7faea4033000 tx=0x7faea40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:06:15.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.716+0000 7fae9bfff640 10 client.?.objecter ms_handle_connect 0x7faeb40ad230 2024-01-21T05:06:15.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.716+0000 7fae9bfff640 10 client.?.objecter resend_mon_ops 2024-01-21T05:06:15.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 1 -- 172.21.15.69:0/3787208009 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea403ac60 con 0x7faeb40ad230 2024-01-21T05:06:15.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 10 monclient: dump: 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 10 monclient: _finish_auth 0 2024-01-21T05:06:15.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 10 monclient: _check_auth_tickets 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 1 -- 172.21.15.69:0/3787208009 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea403adc0 con 0x7faeb40ad230 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7faeb9b97640 5 monclient: authenticate success, global_id 21637 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 1 -- 172.21.15.69:0/3787208009 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea4044630 con 0x7faeb40ad230 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7faeb9b97640 10 monclient: _renew_subs 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7faeb9b97640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7faeb9b97640 1 -- 172.21.15.69:0/3787208009 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb4146390 con 0x7faeb40ad230 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7faeb1ffb640 4 set_mon_vals no callback set 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7fae9bfff640 10 monclient: dump: 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:06:15.723 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7faeb9b97640 10 client.21637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7faeb9b97640 10 monclient: _renew_subs 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7faeb9b97640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7faeb9b97640 1 -- 172.21.15.69:0/3787208009 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb41db3c0 con 0x7faeb40ad230 2024-01-21T05:06:15.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.720+0000 7faeb9b97640 1 librados: init done 2024-01-21T05:06:15.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.721+0000 7faeb9b97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:06:15.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.721+0000 7faeb9b97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:06:15.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.721+0000 7faeb9b97640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:06:15.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.721+0000 7faeb9b97640 1 -- 172.21.15.69:0/3787208009 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb404e970 con 0x7faeb40ad230 2024-01-21T05:06:15.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.964+0000 7fae9bfff640 1 -- 172.21.15.69:0/3787208009 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7faea404d450 con 0x7faeb40ad230 2024-01-21T05:06:15.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.964+0000 7fae9bfff640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7faea006c8b0 0x7faea006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:15.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.964+0000 7faeb2ffd640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7faea006c8b0 0x7faea006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:15.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.964+0000 7faeb2ffd640 10 monclient: get_auth_request con 0x7faea006c8b0 auth_method 0 2024-01-21T05:06:15.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.965+0000 7faeb2ffd640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7faea006c8b0 0x7faea006ed70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7faeb414c280 tx=0x7faea8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:06:15.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.965+0000 7fae9bfff640 10 client.21637.objecter ms_handle_connect 0x7faea006c8b0 2024-01-21T05:06:15.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.966+0000 7fae9bfff640 1 -- 172.21.15.69:0/3787208009 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7faea402d040 con 0x7faeb40ad230 2024-01-21T05:06:15.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.966+0000 7fae9bfff640 10 client.21637.objecter ms_dispatch 0x7faeb4000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:06:15.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.966+0000 7fae9bfff640 3 client.21637.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:06:15.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.966+0000 7fae9bfff640 3 client.21637.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:06:15.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.967+0000 7fae9bfff640 20 client.21637.objecter dump_active .. 0 homeless 2024-01-21T05:06:15.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.976+0000 7fae9bfff640 1 -- 172.21.15.69:0/3787208009 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7faea4076e50 con 0x7faeb40ad230 2024-01-21T05:06:15.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.976+0000 7fae9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:06:15.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:15.976+0000 7fae9bfff640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:06:16.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.151+0000 7faeb9b97640 1 -- 172.21.15.69:0/3787208009 --> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faeb4146210 con 0x7faea006c8b0 2024-01-21T05:06:16.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.156+0000 7fae9bfff640 1 -- 172.21.15.69:0/3787208009 <== mgr.4106 v2:172.21.15.144:6800/1966288294 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151691 (secure 0 0 0) 0x7faea8011a30 con 0x7faea006c8b0 2024-01-21T05:06:16.159 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-01-21T05:06:16.159 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-01-21T05:06:16.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.158+0000 7faeb9b97640 10 librados: watch_flush enter 2024-01-21T05:06:16.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.158+0000 7faeb9b97640 10 librados: watch_flush exit 2024-01-21T05:06:16.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.158+0000 7faeb9b97640 20 client.21637.objecter shutdown clearing up homeless session... 2024-01-21T05:06:16.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.158+0000 7faeb9b97640 10 client.21637.objecter successfully canceled tick 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 -- 172.21.15.69:0/3787208009 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7faea006c8b0 msgr2=0x7faea006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7faea006c8b0 0x7faea006ed70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7faeb414c280 tx=0x7faea8009190 comp rx=0 tx=0).stop 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 10 monclient: shutdown 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 -- 172.21.15.69:0/3787208009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faeb40ad230 msgr2=0x7faeb4135760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faeb40ad230 0x7faeb4135760 secure :-1 s=READY pgs=5243 cs=0 l=1 rev1=1 crypto rx=0x7faea4033000 tx=0x7faea40046c0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 -- 172.21.15.69:0/3787208009 shutdown_connections 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7faea006c8b0 0x7faea006ed70 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faeb4145670 0x7faeb4144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7faeb40b02b0 0x7faeb4144920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 --2- 172.21.15.69:0/3787208009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faeb40ad230 0x7faeb4135760 unknown :-1 s=CLOSED pgs=5243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 -- 172.21.15.69:0/3787208009 >> 172.21.15.69:0/3787208009 conn(0x7faeb40b1130 msgr2=0x7faeb4145b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 -- 172.21.15.69:0/3787208009 shutdown_connections 2024-01-21T05:06:16.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 -- 172.21.15.69:0/3787208009 wait complete. 2024-01-21T05:06:16.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.159+0000 7faeb9b97640 1 librados: shutdown 2024-01-21T05:06:16.170 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":2095,"stamp":"2024-01-21T05:06:16.040891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":606360,"kb_used_data":88984,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754368360,"statfs":{"total":773094113280,"available":772473200640,"internally_reserved":0,"allocated":91119616,"data_stored":49865650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":529714736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002759"},"pg_stats":[{"pgid":"12.1c","version":"0'0","reported_seq":1056,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:55.419700+0000","last_change":"2024-01-21T05:05:55.420490+0000","last_active":"2024-01-21T05:05:55.419700+0000","last_peered":"2024-01-21T05:05:55.419700+0000","last_clean":"2024-01-21T05:05:55.419700+0000","last_became_active":"2024-01-21T04:34:59.874210+0000","last_became_peered":"2024-01-21T04:34:59.874210+0000","last_unstale":"2024-01-21T05:05:55.419700+0000","last_undegraded":"2024-01-21T05:05:55.419700+0000","last_fullsized":"2024-01-21T05:05:55.419700+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:55.418228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:55.418228+0000","last_clean_scrub_stamp":"2024-01-21T05:05:55.418228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:57:30.762813+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1073,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:56.037052+0000","last_change":"2024-01-21T05:05:56.037841+0000","last_active":"2024-01-21T05:05:56.037052+0000","last_peered":"2024-01-21T05:05:56.037052+0000","last_clean":"2024-01-21T05:05:56.037052+0000","last_became_active":"2024-01-21T04:35:19.542355+0000","last_became_peered":"2024-01-21T04:35:19.542355+0000","last_unstale":"2024-01-21T05:05:56.037052+0000","last_undegraded":"2024-01-21T05:05:56.037052+0000","last_fullsized":"2024-01-21T05:05:56.037052+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:56.035530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:56.035530+0000","last_clean_scrub_stamp":"2024-01-21T05:05:56.035530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:31:23.184300+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1096,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:56.109306+0000","last_change":"2024-01-21T05:05:56.110071+0000","last_active":"2024-01-21T05:05:56.109306+0000","last_peered":"2024-01-21T05:05:56.109306+0000","last_clean":"2024-01-21T05:05:56.109306+0000","last_became_active":"2024-01-21T04:35:21.234113+0000","last_became_peered":"2024-01-21T04:35:21.234113+0000","last_unstale":"2024-01-21T05:05:56.109306+0000","last_undegraded":"2024-01-21T05:05:56.109306+0000","last_fullsized":"2024-01-21T05:05:56.109306+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:56.107885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:56.107885+0000","last_clean_scrub_stamp":"2024-01-21T05:05:56.107885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:20:08.428901+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1062,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:59.254545+0000","last_change":"2024-01-21T05:05:59.254545+0000","last_active":"2024-01-21T05:05:59.254545+0000","last_peered":"2024-01-21T05:05:59.254545+0000","last_clean":"2024-01-21T05:05:59.254545+0000","last_became_active":"2024-01-21T04:35:23.463341+0000","last_became_peered":"2024-01-21T04:35:23.463341+0000","last_unstale":"2024-01-21T05:05:59.254545+0000","last_undegraded":"2024-01-21T05:05:59.254545+0000","last_fullsized":"2024-01-21T05:05:59.254545+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:59.252378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:59.252378+0000","last_clean_scrub_stamp":"2024-01-21T05:05:59.252378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:24:36.388830+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1061,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:50.056404+0000","last_change":"2024-01-21T05:05:50.057183+0000","last_active":"2024-01-21T05:05:50.056404+0000","last_peered":"2024-01-21T05:05:50.056404+0000","last_clean":"2024-01-21T05:05:50.056404+0000","last_became_active":"2024-01-21T04:35:16.942042+0000","last_became_peered":"2024-01-21T04:35:16.942042+0000","last_unstale":"2024-01-21T05:05:50.056404+0000","last_undegraded":"2024-01-21T05:05:50.056404+0000","last_fullsized":"2024-01-21T05:05:50.056404+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:50.053903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:50.053903+0000","last_clean_scrub_stamp":"2024-01-21T05:05:50.053903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:08:26.856528+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"1109'8","reported_seq":1089,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:53.146305+0000","last_change":"2024-01-21T05:05:53.146305+0000","last_active":"2024-01-21T05:05:53.146305+0000","last_peered":"2024-01-21T05:05:53.146305+0000","last_clean":"2024-01-21T05:05:53.146305+0000","last_became_active":"2024-01-21T04:35:25.364561+0000","last_became_peered":"2024-01-21T04:35:25.364561+0000","last_unstale":"2024-01-21T05:05:53.146305+0000","last_undegraded":"2024-01-21T05:05:53.146305+0000","last_fullsized":"2024-01-21T05:05:53.146305+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"1109'8","last_scrub_stamp":"2024-01-21T05:05:53.144086+0000","last_deep_scrub":"1109'8","last_deep_scrub_stamp":"2024-01-21T05:05:53.144086+0000","last_clean_scrub_stamp":"2024-01-21T05:05:53.144086+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:50:55.831195+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1096,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:50.978117+0000","last_change":"2024-01-21T05:05:50.978117+0000","last_active":"2024-01-21T05:05:50.978117+0000","last_peered":"2024-01-21T05:05:50.978117+0000","last_clean":"2024-01-21T05:05:50.978117+0000","last_became_active":"2024-01-21T04:34:53.465163+0000","last_became_peered":"2024-01-21T04:34:53.465163+0000","last_unstale":"2024-01-21T05:05:50.978117+0000","last_undegraded":"2024-01-21T05:05:50.978117+0000","last_fullsized":"2024-01-21T05:05:50.978117+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:50.975781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:50.975781+0000","last_clean_scrub_stamp":"2024-01-21T05:05:50.975781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:24:33.611723+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1060,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:54.281919+0000","last_change":"2024-01-21T05:05:54.281919+0000","last_active":"2024-01-21T05:05:54.281919+0000","last_peered":"2024-01-21T05:05:54.281919+0000","last_clean":"2024-01-21T05:05:54.281919+0000","last_became_active":"2024-01-21T04:35:02.595993+0000","last_became_peered":"2024-01-21T04:35:02.595993+0000","last_unstale":"2024-01-21T05:05:54.281919+0000","last_undegraded":"2024-01-21T05:05:54.281919+0000","last_fullsized":"2024-01-21T05:05:54.281919+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:54.279867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:54.279867+0000","last_clean_scrub_stamp":"2024-01-21T05:05:54.279867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:03:20.880672+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1096,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:55.075100+0000","last_change":"2024-01-21T05:05:55.075100+0000","last_active":"2024-01-21T05:05:55.075100+0000","last_peered":"2024-01-21T05:05:55.075100+0000","last_clean":"2024-01-21T05:05:55.075100+0000","last_became_active":"2024-01-21T04:35:21.218816+0000","last_became_peered":"2024-01-21T04:35:21.218816+0000","last_unstale":"2024-01-21T05:05:55.075100+0000","last_undegraded":"2024-01-21T05:05:55.075100+0000","last_fullsized":"2024-01-21T05:05:55.075100+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:55.071524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:55.071524+0000","last_clean_scrub_stamp":"2024-01-21T05:05:55.071524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:29:58.416087+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1059,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:49.049388+0000","last_change":"2024-01-21T05:05:49.050212+0000","last_active":"2024-01-21T05:05:49.049388+0000","last_peered":"2024-01-21T05:05:49.049388+0000","last_clean":"2024-01-21T05:05:49.049388+0000","last_became_active":"2024-01-21T04:34:53.056955+0000","last_became_peered":"2024-01-21T04:34:53.056955+0000","last_unstale":"2024-01-21T05:05:49.049388+0000","last_undegraded":"2024-01-21T05:05:49.049388+0000","last_fullsized":"2024-01-21T05:05:49.049388+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:49.047906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:49.047906+0000","last_clean_scrub_stamp":"2024-01-21T05:05:49.047906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:41:23.348783+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1075,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:53.981981+0000","last_change":"2024-01-21T05:05:53.982875+0000","last_active":"2024-01-21T05:05:53.981981+0000","last_peered":"2024-01-21T05:05:53.981981+0000","last_clean":"2024-01-21T05:05:53.981981+0000","last_became_active":"2024-01-21T04:35:00.838638+0000","last_became_peered":"2024-01-21T04:35:00.838638+0000","last_unstale":"2024-01-21T05:05:53.981981+0000","last_undegraded":"2024-01-21T05:05:53.981981+0000","last_fullsized":"2024-01-21T05:05:53.981981+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:53.979095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:53.979095+0000","last_clean_scrub_stamp":"2024-01-21T05:05:53.979095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:03:00.318714+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1056,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:06:02.995399+0000","last_change":"2024-01-21T05:06:02.996157+0000","last_active":"2024-01-21T05:06:02.995399+0000","last_peered":"2024-01-21T05:06:02.995399+0000","last_clean":"2024-01-21T05:06:02.995399+0000","last_became_active":"2024-01-21T04:35:28.438809+0000","last_became_peered":"2024-01-21T04:35:28.438809+0000","last_unstale":"2024-01-21T05:06:02.995399+0000","last_undegraded":"2024-01-21T05:06:02.995399+0000","last_fullsized":"2024-01-21T05:06:02.995399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:06:02.993901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:06:02.993901+0000","last_clean_scrub_stamp":"2024-01-21T05:06:02.993901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:03:41.331461+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1060,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:57.229784+0000","last_change":"2024-01-21T05:05:57.230544+0000","last_active":"2024-01-21T05:05:57.229784+0000","last_peered":"2024-01-21T05:05:57.229784+0000","last_clean":"2024-01-21T05:05:57.229784+0000","last_became_active":"2024-01-21T04:35:00.824479+0000","last_became_peered":"2024-01-21T04:35:00.824479+0000","last_unstale":"2024-01-21T05:05:57.229784+0000","last_undegraded":"2024-01-21T05:05:57.229784+0000","last_fullsized":"2024-01-21T05:05:57.229784+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:57.228285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:57.228285+0000","last_clean_scrub_stamp":"2024-01-21T05:05:57.228285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:44:09.409405+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"1109'78","reported_seq":1192,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:50.165518+0000","last_change":"2024-01-21T05:05:50.166955+0000","last_active":"2024-01-21T05:05:50.165518+0000","last_peered":"2024-01-21T05:05:50.165518+0000","last_clean":"2024-01-21T05:05:50.165518+0000","last_became_active":"2024-01-21T04:33:33.133732+0000","last_became_peered":"2024-01-21T04:33:33.133732+0000","last_unstale":"2024-01-21T05:05:50.165518+0000","last_undegraded":"2024-01-21T05:05:50.165518+0000","last_fullsized":"2024-01-21T05:05:50.165518+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"1109'78","last_scrub_stamp":"2024-01-21T05:05:50.164075+0000","last_deep_scrub":"1109'78","last_deep_scrub_stamp":"2024-01-21T05:05:50.164075+0000","last_clean_scrub_stamp":"2024-01-21T05:05:50.164075+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:40:51.558576+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1068,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:56.194217+0000","last_change":"2024-01-21T05:05:56.194918+0000","last_active":"2024-01-21T05:05:56.194217+0000","last_peered":"2024-01-21T05:05:56.194217+0000","last_clean":"2024-01-21T05:05:56.194217+0000","last_became_active":"2024-01-21T04:33:48.364387+0000","last_became_peered":"2024-01-21T04:33:48.364387+0000","last_unstale":"2024-01-21T05:05:56.194217+0000","last_undegraded":"2024-01-21T05:05:56.194217+0000","last_fullsized":"2024-01-21T05:05:56.194217+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:56.192879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:56.192879+0000","last_clean_scrub_stamp":"2024-01-21T05:05:56.192879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:18:40.851886+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1056,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:56.915451+0000","last_change":"2024-01-21T05:05:56.916232+0000","last_active":"2024-01-21T05:05:56.915451+0000","last_peered":"2024-01-21T05:05:56.915451+0000","last_clean":"2024-01-21T05:05:56.915451+0000","last_became_active":"2024-01-21T04:34:54.452154+0000","last_became_peered":"2024-01-21T04:34:54.452154+0000","last_unstale":"2024-01-21T05:05:56.915451+0000","last_undegraded":"2024-01-21T05:05:56.915451+0000","last_fullsized":"2024-01-21T05:05:56.915451+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:56.912095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:56.912095+0000","last_clean_scrub_stamp":"2024-01-21T05:05:56.912095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:15:32.061004+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1069,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:52.213786+0000","last_change":"2024-01-21T05:05:52.213786+0000","last_active":"2024-01-21T05:05:52.213786+0000","last_peered":"2024-01-21T05:05:52.213786+0000","last_clean":"2024-01-21T05:05:52.213786+0000","last_became_active":"2024-01-21T04:33:47.734733+0000","last_became_peered":"2024-01-21T04:33:47.734733+0000","last_unstale":"2024-01-21T05:05:52.213786+0000","last_undegraded":"2024-01-21T05:05:52.213786+0000","last_fullsized":"2024-01-21T05:05:52.213786+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:52.211542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:52.211542+0000","last_clean_scrub_stamp":"2024-01-21T05:05:52.211542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T05:50:19.582551+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1060,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:58.276638+0000","last_change":"2024-01-21T05:05:58.277424+0000","last_active":"2024-01-21T05:05:58.276638+0000","last_peered":"2024-01-21T05:05:58.276638+0000","last_clean":"2024-01-21T05:05:58.276638+0000","last_became_active":"2024-01-21T04:35:05.290888+0000","last_became_peered":"2024-01-21T04:35:05.290888+0000","last_unstale":"2024-01-21T05:05:58.276638+0000","last_undegraded":"2024-01-21T05:05:58.276638+0000","last_fullsized":"2024-01-21T05:05:58.276638+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:58.273105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:58.273105+0000","last_clean_scrub_stamp":"2024-01-21T05:05:58.273105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:09:39.460814+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1063,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:56.052682+0000","last_change":"2024-01-21T05:05:56.057060+0000","last_active":"2024-01-21T05:05:56.052682+0000","last_peered":"2024-01-21T05:05:56.052682+0000","last_clean":"2024-01-21T05:05:56.052682+0000","last_became_active":"2024-01-21T04:33:48.670864+0000","last_became_peered":"2024-01-21T04:33:48.670864+0000","last_unstale":"2024-01-21T05:05:56.052682+0000","last_undegraded":"2024-01-21T05:05:56.052682+0000","last_fullsized":"2024-01-21T05:05:56.052682+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:56.048608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:56.048608+0000","last_clean_scrub_stamp":"2024-01-21T05:05:56.048608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:48:06.102424+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1056,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:59.934523+0000","last_change":"2024-01-21T05:05:59.935300+0000","last_active":"2024-01-21T05:05:59.934523+0000","last_peered":"2024-01-21T05:05:59.934523+0000","last_clean":"2024-01-21T05:05:59.934523+0000","last_became_active":"2024-01-21T04:35:07.340277+0000","last_became_peered":"2024-01-21T04:35:07.340277+0000","last_unstale":"2024-01-21T05:05:59.934523+0000","last_undegraded":"2024-01-21T05:05:59.934523+0000","last_fullsized":"2024-01-21T05:05:59.934523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:59.931227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:59.931227+0000","last_clean_scrub_stamp":"2024-01-21T05:05:59.931227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:11:25.974214+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1070,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:54.177220+0000","last_change":"2024-01-21T05:05:54.177936+0000","last_active":"2024-01-21T05:05:54.177220+0000","last_peered":"2024-01-21T05:05:54.177220+0000","last_clean":"2024-01-21T05:05:54.177220+0000","last_became_active":"2024-01-21T04:33:48.015845+0000","last_became_peered":"2024-01-21T04:33:48.015845+0000","last_unstale":"2024-01-21T05:05:54.177220+0000","last_undegraded":"2024-01-21T05:05:54.177220+0000","last_fullsized":"2024-01-21T05:05:54.177220+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:54.175840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:54.175840+0000","last_clean_scrub_stamp":"2024-01-21T05:05:54.175840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:27:20.803105+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1059,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:55.276440+0000","last_change":"2024-01-21T05:05:55.277252+0000","last_active":"2024-01-21T05:05:55.276440+0000","last_peered":"2024-01-21T05:05:55.276440+0000","last_clean":"2024-01-21T05:05:55.276440+0000","last_became_active":"2024-01-21T04:35:16.127034+0000","last_became_peered":"2024-01-21T04:35:16.127034+0000","last_unstale":"2024-01-21T05:05:55.276440+0000","last_undegraded":"2024-01-21T05:05:55.276440+0000","last_fullsized":"2024-01-21T05:05:55.276440+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:55.275058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:55.275058+0000","last_clean_scrub_stamp":"2024-01-21T05:05:55.275058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:54:51.281135+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1064,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:53.481958+0000","last_change":"2024-01-21T05:05:53.482723+0000","last_active":"2024-01-21T05:05:53.481958+0000","last_peered":"2024-01-21T05:05:53.481958+0000","last_clean":"2024-01-21T05:05:53.481958+0000","last_became_active":"2024-01-21T04:33:48.032152+0000","last_became_peered":"2024-01-21T04:33:48.032152+0000","last_unstale":"2024-01-21T05:05:53.481958+0000","last_undegraded":"2024-01-21T05:05:53.481958+0000","last_fullsized":"2024-01-21T05:05:53.481958+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:53.480524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:53.480524+0000","last_clean_scrub_stamp":"2024-01-21T05:05:53.480524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:02:54.200755+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1057,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:57.994603+0000","last_change":"2024-01-21T05:05:57.994603+0000","last_active":"2024-01-21T05:05:57.994603+0000","last_peered":"2024-01-21T05:05:57.994603+0000","last_clean":"2024-01-21T05:05:57.994603+0000","last_became_active":"2024-01-21T04:34:55.527971+0000","last_became_peered":"2024-01-21T04:34:55.527971+0000","last_unstale":"2024-01-21T05:05:57.994603+0000","last_undegraded":"2024-01-21T05:05:57.994603+0000","last_fullsized":"2024-01-21T05:05:57.994603+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:57.992511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:57.992511+0000","last_clean_scrub_stamp":"2024-01-21T05:05:57.992511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:14:00.858659+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1067,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:53.199230+0000","last_change":"2024-01-21T05:05:53.200003+0000","last_active":"2024-01-21T05:05:53.199230+0000","last_peered":"2024-01-21T05:05:53.199230+0000","last_clean":"2024-01-21T05:05:53.199230+0000","last_became_active":"2024-01-21T04:33:47.835639+0000","last_became_peered":"2024-01-21T04:33:47.835639+0000","last_unstale":"2024-01-21T05:05:53.199230+0000","last_undegraded":"2024-01-21T05:05:53.199230+0000","last_fullsized":"2024-01-21T05:05:53.199230+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:53.195526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:53.195526+0000","last_clean_scrub_stamp":"2024-01-21T05:05:53.195526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T07:32:05.517472+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1057,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:06:01.925716+0000","last_change":"2024-01-21T05:06:01.926560+0000","last_active":"2024-01-21T05:06:01.925716+0000","last_peered":"2024-01-21T05:06:01.925716+0000","last_clean":"2024-01-21T05:06:01.925716+0000","last_became_active":"2024-01-21T04:35:26.889439+0000","last_became_peered":"2024-01-21T04:35:26.889439+0000","last_unstale":"2024-01-21T05:06:01.925716+0000","last_undegraded":"2024-01-21T05:06:01.925716+0000","last_fullsized":"2024-01-21T05:06:01.925716+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:06:01.924350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:06:01.924350+0000","last_clean_scrub_stamp":"2024-01-21T05:06:01.924350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:20:25.788290+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1064,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:53.232599+0000","last_change":"2024-01-21T05:05:53.232599+0000","last_active":"2024-01-21T05:05:53.232599+0000","last_peered":"2024-01-21T05:05:53.232599+0000","last_clean":"2024-01-21T05:05:53.232599+0000","last_became_active":"2024-01-21T04:33:48.080898+0000","last_became_peered":"2024-01-21T04:33:48.080898+0000","last_unstale":"2024-01-21T05:05:53.232599+0000","last_undegraded":"2024-01-21T05:05:53.232599+0000","last_fullsized":"2024-01-21T05:05:53.232599+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:53.230449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:53.230449+0000","last_clean_scrub_stamp":"2024-01-21T05:05:53.230449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T12:05:07.276020+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1074,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:57.042583+0000","last_change":"2024-01-21T05:05:57.043351+0000","last_active":"2024-01-21T05:05:57.042583+0000","last_peered":"2024-01-21T05:05:57.042583+0000","last_clean":"2024-01-21T05:05:57.042583+0000","last_became_active":"2024-01-21T04:35:36.901570+0000","last_became_peered":"2024-01-21T04:35:36.901570+0000","last_unstale":"2024-01-21T05:05:57.042583+0000","last_undegraded":"2024-01-21T05:05:57.042583+0000","last_fullsized":"2024-01-21T05:05:57.042583+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:57.041208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:57.041208+0000","last_clean_scrub_stamp":"2024-01-21T05:05:57.041208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:47:50.908618+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1066,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:55.183160+0000","last_change":"2024-01-21T05:05:55.183914+0000","last_active":"2024-01-21T05:05:55.183160+0000","last_peered":"2024-01-21T05:05:55.183160+0000","last_clean":"2024-01-21T05:05:55.183160+0000","last_became_active":"2024-01-21T04:33:48.046404+0000","last_became_peered":"2024-01-21T04:33:48.046404+0000","last_unstale":"2024-01-21T05:05:55.183160+0000","last_undegraded":"2024-01-21T05:05:55.183160+0000","last_fullsized":"2024-01-21T05:05:55.183160+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:55.181742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:55.181742+0000","last_clean_scrub_stamp":"2024-01-21T05:05:55.181742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:47:23.447513+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1058,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:51.113896+0000","last_change":"2024-01-21T05:05:51.114676+0000","last_active":"2024-01-21T05:05:51.113896+0000","last_peered":"2024-01-21T05:05:51.113896+0000","last_clean":"2024-01-21T05:05:51.113896+0000","last_became_active":"2024-01-21T04:35:19.611575+0000","last_became_peered":"2024-01-21T04:35:19.611575+0000","last_unstale":"2024-01-21T05:05:51.113896+0000","last_undegraded":"2024-01-21T05:05:51.113896+0000","last_fullsized":"2024-01-21T05:05:51.113896+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:51.112421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:51.112421+0000","last_clean_scrub_stamp":"2024-01-21T05:05:51.112421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:43:10.316903+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1093,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:52.007025+0000","last_change":"2024-01-21T05:05:52.007785+0000","last_active":"2024-01-21T05:05:52.007025+0000","last_peered":"2024-01-21T05:05:52.007025+0000","last_clean":"2024-01-21T05:05:52.007025+0000","last_became_active":"2024-01-21T04:34:53.495212+0000","last_became_peered":"2024-01-21T04:34:53.495212+0000","last_unstale":"2024-01-21T05:05:52.007025+0000","last_undegraded":"2024-01-21T05:05:52.007025+0000","last_fullsized":"2024-01-21T05:05:52.007025+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:52.005738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:52.005738+0000","last_clean_scrub_stamp":"2024-01-21T05:05:52.005738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T13:59:35.494176+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1055,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:59.000044+0000","last_change":"2024-01-21T05:05:59.000765+0000","last_active":"2024-01-21T05:05:59.000044+0000","last_peered":"2024-01-21T05:05:59.000044+0000","last_clean":"2024-01-21T05:05:59.000044+0000","last_became_active":"2024-01-21T04:34:58.492357+0000","last_became_peered":"2024-01-21T04:34:58.492357+0000","last_unstale":"2024-01-21T05:05:59.000044+0000","last_undegraded":"2024-01-21T05:05:59.000044+0000","last_fullsized":"2024-01-21T05:05:59.000044+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:58.998619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:58.998619+0000","last_clean_scrub_stamp":"2024-01-21T05:05:58.998619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:29:03.709963+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1055,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:06:00.980519+0000","last_change":"2024-01-21T05:06:00.981292+0000","last_active":"2024-01-21T05:06:00.980519+0000","last_peered":"2024-01-21T05:06:00.980519+0000","last_clean":"2024-01-21T05:06:00.980519+0000","last_became_active":"2024-01-21T04:35:25.928650+0000","last_became_peered":"2024-01-21T04:35:25.928650+0000","last_unstale":"2024-01-21T05:06:00.980519+0000","last_undegraded":"2024-01-21T05:06:00.980519+0000","last_fullsized":"2024-01-21T05:06:00.980519+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:06:00.977148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:06:00.977148+0000","last_clean_scrub_stamp":"2024-01-21T05:06:00.977148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T14:45:20.599770+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1072,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:51.076628+0000","last_change":"2024-01-21T05:05:51.077414+0000","last_active":"2024-01-21T05:05:51.076628+0000","last_peered":"2024-01-21T05:05:51.076628+0000","last_clean":"2024-01-21T05:05:51.076628+0000","last_became_active":"2024-01-21T04:35:00.387495+0000","last_became_peered":"2024-01-21T04:35:00.387495+0000","last_unstale":"2024-01-21T05:05:51.076628+0000","last_undegraded":"2024-01-21T05:05:51.076628+0000","last_fullsized":"2024-01-21T05:05:51.076628+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:51.075137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:51.075137+0000","last_clean_scrub_stamp":"2024-01-21T05:05:51.075137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T08:10:50.701439+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1094,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:54.066478+0000","last_change":"2024-01-21T05:05:54.066478+0000","last_active":"2024-01-21T05:05:54.066478+0000","last_peered":"2024-01-21T05:05:54.066478+0000","last_clean":"2024-01-21T05:05:54.066478+0000","last_became_active":"2024-01-21T04:35:16.679256+0000","last_became_peered":"2024-01-21T04:35:16.679256+0000","last_unstale":"2024-01-21T05:05:54.066478+0000","last_undegraded":"2024-01-21T05:05:54.066478+0000","last_fullsized":"2024-01-21T05:05:54.066478+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:54.062875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:54.062875+0000","last_clean_scrub_stamp":"2024-01-21T05:05:54.062875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T15:17:56.645863+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1073,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:52.081900+0000","last_change":"2024-01-21T05:05:52.081900+0000","last_active":"2024-01-21T05:05:52.081900+0000","last_peered":"2024-01-21T05:05:52.081900+0000","last_clean":"2024-01-21T05:05:52.081900+0000","last_became_active":"2024-01-21T04:35:25.004049+0000","last_became_peered":"2024-01-21T04:35:25.004049+0000","last_unstale":"2024-01-21T05:05:52.081900+0000","last_undegraded":"2024-01-21T05:05:52.081900+0000","last_fullsized":"2024-01-21T05:05:52.081900+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:52.079749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:52.079749+0000","last_clean_scrub_stamp":"2024-01-21T05:05:52.079749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:44:31.431778+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1059,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:52.072220+0000","last_change":"2024-01-21T05:05:52.072962+0000","last_active":"2024-01-21T05:05:52.072220+0000","last_peered":"2024-01-21T05:05:52.072220+0000","last_clean":"2024-01-21T05:05:52.072220+0000","last_became_active":"2024-01-21T04:35:23.567833+0000","last_became_peered":"2024-01-21T04:35:23.567833+0000","last_unstale":"2024-01-21T05:05:52.072220+0000","last_undegraded":"2024-01-21T05:05:52.072220+0000","last_fullsized":"2024-01-21T05:05:52.072220+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:52.070806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:52.070806+0000","last_clean_scrub_stamp":"2024-01-21T05:05:52.070806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T06:53:40.745056+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1057,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:54.543412+0000","last_change":"2024-01-21T05:05:54.544202+0000","last_active":"2024-01-21T05:05:54.543412+0000","last_peered":"2024-01-21T05:05:54.543412+0000","last_clean":"2024-01-21T05:05:54.543412+0000","last_became_active":"2024-01-21T04:34:57.153977+0000","last_became_peered":"2024-01-21T04:34:57.153977+0000","last_unstale":"2024-01-21T05:05:54.543412+0000","last_undegraded":"2024-01-21T05:05:54.543412+0000","last_fullsized":"2024-01-21T05:05:54.543412+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:54.541963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:54.541963+0000","last_clean_scrub_stamp":"2024-01-21T05:05:54.541963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T10:52:54.555296+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1058,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:56.306610+0000","last_change":"2024-01-21T05:05:56.307330+0000","last_active":"2024-01-21T05:05:56.306610+0000","last_peered":"2024-01-21T05:05:56.306610+0000","last_clean":"2024-01-21T05:05:56.306610+0000","last_became_active":"2024-01-21T04:35:28.583632+0000","last_became_peered":"2024-01-21T04:35:28.583632+0000","last_unstale":"2024-01-21T05:05:56.306610+0000","last_undegraded":"2024-01-21T05:05:56.306610+0000","last_fullsized":"2024-01-21T05:05:56.306610+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:56.305282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:56.305282+0000","last_clean_scrub_stamp":"2024-01-21T05:05:56.305282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T09:47:57.122784+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1093,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:53.036057+0000","last_change":"2024-01-21T05:05:53.036817+0000","last_active":"2024-01-21T05:05:53.036057+0000","last_peered":"2024-01-21T05:05:53.036057+0000","last_clean":"2024-01-21T05:05:53.036057+0000","last_became_active":"2024-01-21T04:35:16.734793+0000","last_became_peered":"2024-01-21T04:35:16.734793+0000","last_unstale":"2024-01-21T05:05:53.036057+0000","last_undegraded":"2024-01-21T05:05:53.036057+0000","last_fullsized":"2024-01-21T05:05:53.036057+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:53.034619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:53.034619+0000","last_clean_scrub_stamp":"2024-01-21T05:05:53.034619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T11:19:35.298063+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1074,"reported_epoch":1109,"state":"active+clean","last_fresh":"2024-01-21T05:05:55.011355+0000","last_change":"2024-01-21T05:05:55.012104+0000","last_active":"2024-01-21T05:05:55.011355+0000","last_peered":"2024-01-21T05:05:55.011355+0000","last_clean":"2024-01-21T05:05:55.011355+0000","last_became_active":"2024-01-21T04:35:14.685816+0000","last_became_peered":"2024-01-21T04:35:14.685816+0000","last_unstale":"2024-01-21T05:05:55.011355+0000","last_undegraded":"2024-01-21T05:05:55.011355+0000","last_fullsized":"2024-01-21T05:05:55.011355+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-21T05:05:55.009858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-21T05:05:55.009858+0000","last_clean_scrub_stamp":"2024-01-21T05:05:55.009858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-22T16:19:56.512930+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":21,"seq":90194313610,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74016,"kb_used_data":10904,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297824,"statfs":{"total":96636764160,"available":96560971776,"internally_reserved":0,"allocated":11165696,"data_stored":6053593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:06:03 2024","interfaces":[{"interface":"back","average":{"1min":17.242999999999999,"5min":12.561,"15min":16.989999999999998},"min":{"1min":6.7160000000000002,"5min":5.4320000000000004,"15min":4.5259999999999998},"max":{"1min":98.305000000000007,"5min":130.69499999999999,"15min":305.86900000000003},"last":8.3480000000000008},{"interface":"front","average":{"1min":21.213999999999999,"5min":15.884,"15min":20.588999999999999},"min":{"1min":8.6159999999999997,"5min":8.0370000000000008,"15min":4.6710000000000003},"max":{"1min":75.066000000000003,"5min":121.011,"15min":281.79000000000002},"last":9.8309999999999995}]},{"osd":1,"last update":"Sun Jan 21 05:05:28 2024","interfaces":[{"interface":"back","average":{"1min":30.696999999999999,"5min":16.850999999999999,"15min":19.510000000000002},"min":{"1min":9.0579999999999998,"5min":7.6550000000000002,"15min":6.1280000000000001},"max":{"1min":151.977,"5min":151.977,"15min":249.82599999999999},"last":15.093},{"interface":"front","average":{"1min":21.411999999999999,"5min":10.917,"15min":14.314},"min":{"1min":5.5709999999999997,"5min":4.8860000000000001,"15min":4.4429999999999996},"max":{"1min":133.238,"5min":133.238,"15min":185.57599999999999},"last":7.9809999999999999}]},{"osd":2,"last update":"Sun Jan 21 05:05:16 2024","interfaces":[{"interface":"back","average":{"1min":18.077000000000002,"5min":11.068,"15min":16.349},"min":{"1min":6.9770000000000003,"5min":6.4000000000000004,"15min":6.4000000000000004},"max":{"1min":102.258,"5min":102.258,"15min":260.14999999999998},"last":9.2119999999999997},{"interface":"front","average":{"1min":24.271000000000001,"5min":15.379,"15min":22.163},"min":{"1min":9.6739999999999995,"5min":7.8390000000000004,"15min":7.8390000000000004},"max":{"1min":108.218,"5min":108.218,"15min":261.67200000000003},"last":14.398}]},{"osd":4,"last update":"Sun Jan 21 05:06:07 2024","interfaces":[{"interface":"back","average":{"1min":19.315000000000001,"5min":14.709,"15min":16.766999999999999},"min":{"1min":7.3250000000000002,"5min":6.1299999999999999,"15min":6.0830000000000002},"max":{"1min":121.16200000000001,"5min":173.92099999999999,"15min":179.381},"last":10.506},{"interface":"front","average":{"1min":21.427,"5min":16.988,"15min":19.190999999999999},"min":{"1min":7.077,"5min":7.077,"15min":6.766},"max":{"1min":145.89599999999999,"5min":145.89599999999999,"15min":209.67099999999999},"last":11.853999999999999}]},{"osd":5,"last update":"Sun Jan 21 05:06:00 2024","interfaces":[{"interface":"back","average":{"1min":24.869,"5min":17.149000000000001,"15min":22.902000000000001},"min":{"1min":8.9619999999999997,"5min":7.8719999999999999,"15min":6.9800000000000004},"max":{"1min":109.703,"5min":109.703,"15min":318.983},"last":11.177},{"interface":"front","average":{"1min":42.771999999999998,"5min":18.417000000000002,"15min":22.704999999999998},"min":{"1min":8.3030000000000008,"5min":7.3419999999999996,"15min":6.7939999999999996},"max":{"1min":248.768,"5min":248.768,"15min":341.21699999999998},"last":12.502000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:06:03 2024","interfaces":[{"interface":"back","average":{"1min":28.475999999999999,"5min":19.847999999999999,"15min":25.027999999999999},"min":{"1min":10.662000000000001,"5min":10.154999999999999,"15min":7.1859999999999999},"max":{"1min":187.625,"5min":187.625,"15min":265.88999999999999},"last":16.474},{"interface":"front","average":{"1min":36.036999999999999,"5min":23.079999999999998,"15min":26.800999999999998},"min":{"1min":12.603999999999999,"5min":11.544,"15min":11.026},"max":{"1min":312.58100000000002,"5min":312.58100000000002,"15min":312.58100000000002},"last":20.279}]},{"osd":7,"last update":"Sun Jan 21 05:05:28 2024","interfaces":[{"interface":"back","average":{"1min":40.505000000000003,"5min":19.132000000000001,"15min":21.704000000000001},"min":{"1min":9.1530000000000005,"5min":7.8639999999999999,"15min":7.7240000000000002},"max":{"1min":233.38499999999999,"5min":233.38499999999999,"15min":233.38499999999999},"last":15.778},{"interface":"front","average":{"1min":41.484999999999999,"5min":21.462,"15min":23.84},"min":{"1min":9.9809999999999999,"5min":9.9120000000000008,"15min":9.9120000000000008},"max":{"1min":264.34800000000001,"5min":264.34800000000001,"15min":264.34800000000001},"last":18.170999999999999}]}]},{"osd":6,"up_from":20,"seq":85899346314,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74260,"kb_used_data":11148,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297580,"statfs":{"total":96636764160,"available":96560721920,"internally_reserved":0,"allocated":11415552,"data_stored":6213902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":21.414000000000001,"5min":10.807,"15min":18.202999999999999},"min":{"1min":5.2009999999999996,"5min":4.109,"15min":4.0830000000000002},"max":{"1min":170.822,"5min":170.822,"15min":293.416},"last":6.1909999999999998},{"interface":"front","average":{"1min":23.597000000000001,"5min":11.395,"15min":18.113},"min":{"1min":5.6059999999999999,"5min":4.3849999999999998,"15min":4.2220000000000004},"max":{"1min":170.15700000000001,"5min":170.15700000000001,"15min":258.678},"last":7.8810000000000002}]},{"osd":1,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":22.026,"5min":13.266,"15min":17.341000000000001},"min":{"1min":7.5270000000000001,"5min":6.2839999999999998,"15min":6.0650000000000004},"max":{"1min":75.444999999999993,"5min":75.444999999999993,"15min":267.21899999999999},"last":12.17},{"interface":"front","average":{"1min":31.890999999999998,"5min":17.786000000000001,"15min":22.844999999999999},"min":{"1min":10.311999999999999,"5min":8.6660000000000004,"15min":7.3339999999999996},"max":{"1min":151.76499999999999,"5min":151.76499999999999,"15min":271.875},"last":14.327}]},{"osd":2,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":23.411999999999999,"5min":13.505000000000001,"15min":20.018000000000001},"min":{"1min":9.3260000000000005,"5min":6.0039999999999996,"15min":6.0039999999999996},"max":{"1min":63.945999999999998,"5min":63.945999999999998,"15min":274.54000000000002},"last":10.977},{"interface":"front","average":{"1min":24.706,"5min":16.004000000000001,"15min":18.666},"min":{"1min":11.185,"5min":7.5869999999999997,"15min":5.9420000000000002},"max":{"1min":71.260999999999996,"5min":71.260999999999996,"15min":163.05199999999999},"last":17.149999999999999}]},{"osd":3,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":32.323,"5min":16.699000000000002,"15min":18.744},"min":{"1min":8.1890000000000001,"5min":6.7919999999999998,"15min":6.7549999999999999},"max":{"1min":162.52799999999999,"5min":162.52799999999999,"15min":162.52799999999999},"last":11.846},{"interface":"front","average":{"1min":26.635999999999999,"5min":17.824000000000002,"15min":22.774999999999999},"min":{"1min":11.757,"5min":10.016,"15min":8.2469999999999999},"max":{"1min":72.900999999999996,"5min":72.900999999999996,"15min":275.81},"last":17.742999999999999}]},{"osd":4,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":30.867999999999999,"5min":19.631,"15min":25.166},"min":{"1min":12.084,"5min":11.298,"15min":9.625},"max":{"1min":128.995,"5min":128.995,"15min":280.101},"last":18.798999999999999},{"interface":"front","average":{"1min":32.468000000000004,"5min":21.239000000000001,"15min":22.986000000000001},"min":{"1min":14.935,"5min":11.999000000000001,"15min":6.4119999999999999},"max":{"1min":127.55,"5min":127.55,"15min":273.45600000000002},"last":19.573}]},{"osd":5,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":23.670999999999999,"5min":13.929,"15min":21.393999999999998},"min":{"1min":6.8079999999999998,"5min":6.1699999999999999,"15min":6.1699999999999999},"max":{"1min":98.465999999999994,"5min":98.465999999999994,"15min":275.233},"last":15.332000000000001},{"interface":"front","average":{"1min":26.530000000000001,"5min":16.762,"15min":22.911999999999999},"min":{"1min":10.563000000000001,"5min":7.9710000000000001,"15min":7.0179999999999998},"max":{"1min":66.147000000000006,"5min":66.147000000000006,"15min":288.76499999999999},"last":13.558999999999999}]},{"osd":7,"last update":"Sun Jan 21 05:05:52 2024","interfaces":[{"interface":"back","average":{"1min":24.048999999999999,"5min":14.968999999999999,"15min":22.209},"min":{"1min":8.3249999999999993,"5min":7.5220000000000002,"15min":7.5220000000000002},"max":{"1min":73.034000000000006,"5min":73.034000000000006,"15min":366.68000000000001},"last":10.705},{"interface":"front","average":{"1min":26.388000000000002,"5min":17.59,"15min":23.196999999999999},"min":{"1min":10.593,"5min":9.6989999999999998,"15min":9.6790000000000003},"max":{"1min":74.290999999999997,"5min":74.290999999999997,"15min":281.78500000000003},"last":15.983000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346315,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74016,"kb_used_data":10904,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297824,"statfs":{"total":96636764160,"available":96560971776,"internally_reserved":0,"allocated":11165696,"data_stored":6053593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:06:00 2024","interfaces":[{"interface":"back","average":{"1min":21.907,"5min":13.397,"15min":17.122},"min":{"1min":5.0350000000000001,"5min":5.0350000000000001,"15min":4.5709999999999997},"max":{"1min":157.75800000000001,"5min":157.75800000000001,"15min":294.92099999999999},"last":10.345000000000001},{"interface":"front","average":{"1min":21.039999999999999,"5min":16.181999999999999,"15min":19.477},"min":{"1min":7.9669999999999996,"5min":7.7859999999999996,"15min":4.6699999999999999},"max":{"1min":118.97,"5min":118.97,"15min":292.43700000000001},"last":17.677}]},{"osd":1,"last update":"Sun Jan 21 05:06:00 2024","interfaces":[{"interface":"back","average":{"1min":14.939,"5min":11.006,"15min":15.526},"min":{"1min":6.343,"5min":6.0350000000000001,"15min":4.1470000000000002},"max":{"1min":62.765000000000001,"5min":62.765000000000001,"15min":221.84999999999999},"last":7.7149999999999999},{"interface":"front","average":{"1min":18.780999999999999,"5min":14.598000000000001,"15min":18.16},"min":{"1min":9.4540000000000006,"5min":7.2190000000000003,"15min":6.3959999999999999},"max":{"1min":70.772000000000006,"5min":70.772000000000006,"15min":299.11599999999999},"last":8.4149999999999991}]},{"osd":2,"last update":"Sun Jan 21 05:06:00 2024","interfaces":[{"interface":"back","average":{"1min":36.494,"5min":18.462,"15min":21.265000000000001},"min":{"1min":9.4139999999999997,"5min":7.0209999999999999,"15min":5.7439999999999998},"max":{"1min":404.51299999999998,"5min":404.51299999999998,"15min":404.51299999999998},"last":15.375},{"interface":"front","average":{"1min":22.530000000000001,"5min":13.686999999999999,"15min":20.545000000000002},"min":{"1min":7.2729999999999997,"5min":6.4829999999999997,"15min":6.4829999999999997},"max":{"1min":183.035,"5min":183.035,"15min":303.94799999999998},"last":11.295}]},{"osd":3,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":29.370000000000001,"5min":16.681000000000001,"15min":22.192},"min":{"1min":6.0940000000000003,"5min":6.0940000000000003,"15min":6.0940000000000003},"max":{"1min":226.172,"5min":226.172,"15min":340.21100000000001},"last":13.124000000000001},{"interface":"front","average":{"1min":36.143999999999998,"5min":16.957999999999998,"15min":22.98},"min":{"1min":9.0310000000000006,"5min":6.6529999999999996,"15min":6.5579999999999998},"max":{"1min":211.38300000000001,"5min":211.38300000000001,"15min":302.72500000000002},"last":14.672000000000001}]},{"osd":5,"last update":"Sun Jan 21 05:06:00 2024","interfaces":[{"interface":"back","average":{"1min":22.427,"5min":17.516999999999999,"15min":23.887},"min":{"1min":8.0709999999999997,"5min":7.2510000000000003,"15min":7.2510000000000003},"max":{"1min":95.043000000000006,"5min":206.16800000000001,"15min":305.17200000000003},"last":12.467000000000001},{"interface":"front","average":{"1min":24.161999999999999,"5min":20.286999999999999,"15min":28.995999999999999},"min":{"1min":11.699999999999999,"5min":9.9359999999999999,"15min":9.9359999999999999},"max":{"1min":86.567999999999998,"5min":192.066,"15min":326.19900000000001},"last":13.898999999999999}]},{"osd":6,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":24.899999999999999,"5min":15.957000000000001,"15min":23.010999999999999},"min":{"1min":6.8769999999999998,"5min":6.7409999999999997,"15min":6.7409999999999997},"max":{"1min":204.32900000000001,"5min":204.32900000000001,"15min":248.09800000000001},"last":17.024999999999999},{"interface":"front","average":{"1min":27.047000000000001,"5min":18.641999999999999,"15min":26.821000000000002},"min":{"1min":7.7800000000000002,"5min":7.7800000000000002,"15min":7.7800000000000002},"max":{"1min":205.94800000000001,"5min":205.94800000000001,"15min":333.46899999999999},"last":18.504999999999999}]},{"osd":7,"last update":"Sun Jan 21 05:05:39 2024","interfaces":[{"interface":"back","average":{"1min":28.289999999999999,"5min":17.347000000000001,"15min":20.901},"min":{"1min":9.5459999999999994,"5min":7.3970000000000002,"15min":6.5430000000000001},"max":{"1min":260.63499999999999,"5min":260.63499999999999,"15min":312.41199999999998},"last":16.07},{"interface":"front","average":{"1min":29.556000000000001,"5min":19.172999999999998,"15min":25.035},"min":{"1min":8.3840000000000003,"5min":8.3840000000000003,"15min":8.3840000000000003},"max":{"1min":236.80799999999999,"5min":236.80799999999999,"15min":247.44499999999999},"last":20.786999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509835,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74256,"kb_used_data":11144,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297584,"statfs":{"total":96636764160,"available":96560726016,"internally_reserved":0,"allocated":11411456,"data_stored":6213897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:21 2024","interfaces":[{"interface":"back","average":{"1min":26.605,"5min":13.619999999999999,"15min":20.289000000000001},"min":{"1min":4.9279999999999999,"5min":4.46,"15min":3.847},"max":{"1min":244.893,"5min":244.893,"15min":272.279},"last":6.423},{"interface":"front","average":{"1min":27.978000000000002,"5min":12.045999999999999,"15min":19.513000000000002},"min":{"1min":4.516,"5min":4.0579999999999998,"15min":4.0579999999999998},"max":{"1min":224.22800000000001,"5min":224.22800000000001,"15min":356.99799999999999},"last":10.782}]},{"osd":1,"last update":"Sun Jan 21 05:05:55 2024","interfaces":[{"interface":"back","average":{"1min":25.760999999999999,"5min":12.327999999999999,"15min":15.638},"min":{"1min":5.7030000000000003,"5min":4.7889999999999997,"15min":4.7759999999999998},"max":{"1min":205.53899999999999,"5min":205.53899999999999,"15min":276.94099999999997},"last":14.715999999999999},{"interface":"front","average":{"1min":26.640999999999998,"5min":15.640000000000001,"15min":20.829999999999998},"min":{"1min":8.3290000000000006,"5min":7.4729999999999999,"15min":6.383},"max":{"1min":247.072,"5min":247.072,"15min":247.072},"last":15.448}]},{"osd":2,"last update":"Sun Jan 21 05:05:21 2024","interfaces":[{"interface":"back","average":{"1min":32.673999999999999,"5min":17.591000000000001,"15min":25.193000000000001},"min":{"1min":8.125,"5min":7.5919999999999996,"15min":7.5919999999999996},"max":{"1min":259.29899999999998,"5min":259.29899999999998,"15min":284.34100000000001},"last":14.065},{"interface":"front","average":{"1min":35.651000000000003,"5min":20.353000000000002,"15min":24.027999999999999},"min":{"1min":14.430999999999999,"5min":11.855,"15min":6.0229999999999997},"max":{"1min":260.87799999999999,"5min":260.87799999999999,"15min":260.87799999999999},"last":19.106999999999999}]},{"osd":3,"last update":"Sun Jan 21 05:05:37 2024","interfaces":[{"interface":"back","average":{"1min":41.347000000000001,"5min":21.199000000000002,"15min":26.056000000000001},"min":{"1min":8.8219999999999992,"5min":8.7919999999999998,"15min":7.8109999999999999},"max":{"1min":263.053,"5min":263.053,"15min":276.61399999999998},"last":17.513000000000002},{"interface":"front","average":{"1min":44.539999999999999,"5min":23.657,"15min":29.605},"min":{"1min":14.435,"5min":13.215,"15min":10.497},"max":{"1min":265.154,"5min":265.154,"15min":277.28100000000001},"last":19.731000000000002}]},{"osd":4,"last update":"Sun Jan 21 05:05:37 2024","interfaces":[{"interface":"back","average":{"1min":50.813000000000002,"5min":18.835000000000001,"15min":17.899000000000001},"min":{"1min":7.5730000000000004,"5min":5.8620000000000001,"15min":5.8620000000000001},"max":{"1min":316.40699999999998,"5min":316.40699999999998,"15min":316.40699999999998},"last":8.1170000000000009},{"interface":"front","average":{"1min":45.965000000000003,"5min":23.863,"15min":25.629000000000001},"min":{"1min":15.303000000000001,"5min":12.269,"15min":9.1609999999999996},"max":{"1min":256.67099999999999,"5min":256.67099999999999,"15min":263.23099999999999},"last":18.298999999999999}]},{"osd":5,"last update":"Sun Jan 21 05:05:22 2024","interfaces":[{"interface":"back","average":{"1min":38.140999999999998,"5min":16.295000000000002,"15min":21.081},"min":{"1min":8.4489999999999998,"5min":6.6459999999999999,"15min":6.6429999999999998},"max":{"1min":223.56999999999999,"5min":223.56999999999999,"15min":223.56999999999999},"last":12.093999999999999},{"interface":"front","average":{"1min":40.212000000000003,"5min":19.149000000000001,"15min":23.727},"min":{"1min":8.6560000000000006,"5min":8.6240000000000006,"15min":8.5429999999999993},"max":{"1min":224.81100000000001,"5min":224.81100000000001,"15min":224.81100000000001},"last":12.787000000000001}]},{"osd":6,"last update":"Sun Jan 21 05:05:55 2024","interfaces":[{"interface":"back","average":{"1min":37.509999999999998,"5min":18.471,"15min":26.765999999999998},"min":{"1min":7.1420000000000003,"5min":7.1420000000000003,"15min":7.1420000000000003},"max":{"1min":225.74299999999999,"5min":225.74299999999999,"15min":371.17599999999999},"last":16.131},{"interface":"front","average":{"1min":26.777000000000001,"5min":16.038,"15min":19.922000000000001},"min":{"1min":9.9450000000000003,"5min":7.5679999999999996,"15min":7.1159999999999997},"max":{"1min":214.459,"5min":214.459,"15min":214.459},"last":11.177}]}]},{"osd":0,"up_from":15,"seq":64424509831,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78176,"kb_used_data":10904,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94293664,"statfs":{"total":96636764160,"available":96556711936,"internally_reserved":0,"allocated":11165696,"data_stored":6053593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":56.753,"5min":18.943000000000001,"15min":22.891999999999999},"min":{"1min":5.2919999999999998,"5min":4.8209999999999997,"15min":4.5629999999999997},"max":{"1min":310.80200000000002,"5min":310.80200000000002,"15min":310.80200000000002},"last":9.9990000000000006},{"interface":"front","average":{"1min":46.957999999999998,"5min":16.795999999999999,"15min":20.858000000000001},"min":{"1min":7.165,"5min":4.9880000000000004,"15min":4.9880000000000004},"max":{"1min":303.25200000000001,"5min":303.25200000000001,"15min":303.25200000000001},"last":5.7229999999999999}]},{"osd":2,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":40.720999999999997,"5min":16.710999999999999,"15min":23.585000000000001},"min":{"1min":6.7960000000000003,"5min":6.2800000000000002,"15min":5.5469999999999997},"max":{"1min":275.13299999999998,"5min":275.13299999999998,"15min":404.36700000000002},"last":11.231},{"interface":"front","average":{"1min":47.176000000000002,"5min":21.809999999999999,"15min":30.907},"min":{"1min":9.7240000000000002,"5min":9.4169999999999998,"15min":8.8650000000000002},"max":{"1min":280.69900000000001,"5min":280.69900000000001,"15min":407.68000000000001},"last":18.167000000000002}]},{"osd":3,"last update":"Sun Jan 21 05:06:08 2024","interfaces":[{"interface":"back","average":{"1min":27.268000000000001,"5min":19.963000000000001,"15min":24.969999999999999},"min":{"1min":6.6689999999999996,"5min":6.6689999999999996,"15min":6.266},"max":{"1min":191.05099999999999,"5min":212.34200000000001,"15min":253.25399999999999},"last":8.6229999999999993},{"interface":"front","average":{"1min":16.917999999999999,"5min":11.685,"15min":17.253},"min":{"1min":6.319,"5min":5.7930000000000001,"15min":5.7930000000000001},"max":{"1min":91.926000000000002,"5min":91.926000000000002,"15min":315.92399999999998},"last":17.300000000000001}]},{"osd":4,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":27.547000000000001,"5min":14.999000000000001,"15min":20.422999999999998},"min":{"1min":6.7510000000000003,"5min":6.7510000000000003,"15min":6.2089999999999996},"max":{"1min":95.712999999999994,"5min":95.712999999999994,"15min":283.37599999999998},"last":9.2449999999999992},{"interface":"front","average":{"1min":56.161999999999999,"5min":24.68,"15min":30.161000000000001},"min":{"1min":12.042999999999999,"5min":9.7360000000000007,"15min":9.7360000000000007},"max":{"1min":336.72899999999998,"5min":336.72899999999998,"15min":344.37200000000001},"last":18.969999999999999}]},{"osd":5,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":32.240000000000002,"5min":17.824000000000002,"15min":26.983000000000001},"min":{"1min":9.1850000000000005,"5min":7.4290000000000003,"15min":6.6029999999999998},"max":{"1min":211.78800000000001,"5min":211.78800000000001,"15min":354.04300000000001},"last":12.135},{"interface":"front","average":{"1min":27.254999999999999,"5min":15.403,"15min":23.954000000000001},"min":{"1min":8.7759999999999998,"5min":7.7089999999999996,"15min":6.3159999999999998},"max":{"1min":192.12899999999999,"5min":192.12899999999999,"15min":254.476},"last":12.769}]},{"osd":6,"last update":"Sun Jan 21 05:06:02 2024","interfaces":[{"interface":"back","average":{"1min":35.180999999999997,"5min":20.649999999999999,"15min":23.667999999999999},"min":{"1min":11.208,"5min":8.1799999999999997,"15min":7.4969999999999999},"max":{"1min":208.50299999999999,"5min":226.14599999999999,"15min":256.55900000000003},"last":13.401999999999999},{"interface":"front","average":{"1min":37.533000000000001,"5min":23.372,"15min":28.350000000000001},"min":{"1min":13.145,"5min":10.058,"15min":8.8550000000000004},"max":{"1min":204.67099999999999,"5min":236.083,"15min":258.23899999999998},"last":16.338999999999999}]},{"osd":7,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":28.417999999999999,"5min":15.923999999999999,"15min":23.308},"min":{"1min":9.2400000000000002,"5min":8.0700000000000003,"15min":7.6509999999999998},"max":{"1min":113.319,"5min":113.319,"15min":329.11200000000002},"last":14.15},{"interface":"front","average":{"1min":33.444000000000003,"5min":19.013000000000002,"15min":28.425999999999998},"min":{"1min":10.021000000000001,"5min":10.021000000000001,"15min":9.3699999999999992},"max":{"1min":115.97,"5min":115.97,"15min":337.24000000000001},"last":14.858000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509833,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82336,"kb_used_data":10904,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94289504,"statfs":{"total":96636764160,"available":96552452096,"internally_reserved":0,"allocated":11165696,"data_stored":6053593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":14.563000000000001,"5min":11.273999999999999,"15min":15.429},"min":{"1min":4.4800000000000004,"5min":4.4800000000000004,"15min":4.343},"max":{"1min":68.804000000000002,"5min":68.804000000000002,"15min":188.334},"last":10.188000000000001},{"interface":"front","average":{"1min":10.598000000000001,"5min":9.1699999999999999,"15min":14.536},"min":{"1min":5.8540000000000001,"5min":5.0380000000000003,"15min":4.2919999999999998},"max":{"1min":26.100000000000001,"5min":26.100000000000001,"15min":153.12299999999999},"last":7.6820000000000004}]},{"osd":1,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":13.564,"5min":10.316000000000001,"15min":16.317},"min":{"1min":5.6440000000000001,"5min":5.5419999999999998,"15min":5.5419999999999998},"max":{"1min":37.771999999999998,"5min":37.771999999999998,"15min":318.404},"last":9.5980000000000008},{"interface":"front","average":{"1min":19.129000000000001,"5min":14.199,"15min":21.916},"min":{"1min":10.805999999999999,"5min":7.1699999999999999,"15min":7.1699999999999999},"max":{"1min":58.195,"5min":58.195,"15min":333.91000000000003},"last":15.121}]},{"osd":3,"last update":"Sun Jan 21 05:05:24 2024","interfaces":[{"interface":"back","average":{"1min":17.065000000000001,"5min":12.718,"15min":18.966999999999999},"min":{"1min":7.1639999999999997,"5min":6.9340000000000002,"15min":5.7240000000000002},"max":{"1min":48.807000000000002,"5min":48.807000000000002,"15min":314.58600000000001},"last":11.385},{"interface":"front","average":{"1min":19.007000000000001,"5min":15.404999999999999,"15min":22.751000000000001},"min":{"1min":9.9469999999999992,"5min":9.7530000000000001,"15min":8.5250000000000004},"max":{"1min":71.736999999999995,"5min":71.736999999999995,"15min":335.71699999999998},"last":10.801}]},{"osd":4,"last update":"Sun Jan 21 05:05:43 2024","interfaces":[{"interface":"back","average":{"1min":19.684000000000001,"5min":14.772,"15min":20.323},"min":{"1min":9.1600000000000001,"5min":8.25,"15min":6.0999999999999996},"max":{"1min":54.728999999999999,"5min":54.728999999999999,"15min":313.17200000000003},"last":12.795999999999999},{"interface":"front","average":{"1min":21.738,"5min":17.106000000000002,"15min":22.914999999999999},"min":{"1min":12.092000000000001,"5min":9.593,"15min":7.6070000000000002},"max":{"1min":75.061999999999998,"5min":75.061999999999998,"15min":331.18200000000002},"last":12.111000000000001}]},{"osd":5,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":20.053999999999998,"5min":12.343999999999999,"15min":19.91},"min":{"1min":6.8529999999999998,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":166.327,"5min":166.327,"15min":270.91399999999999},"last":8.593},{"interface":"front","average":{"1min":24.475999999999999,"5min":17.946000000000002,"15min":24.292999999999999},"min":{"1min":13.491,"5min":10.361000000000001,"15min":8.8279999999999994},"max":{"1min":130.548,"5min":130.548,"15min":259.60399999999998},"last":15.721}]},{"osd":6,"last update":"Sun Jan 21 05:05:54 2024","interfaces":[{"interface":"back","average":{"1min":25.367999999999999,"5min":19.186,"15min":26.861999999999998},"min":{"1min":13.91,"5min":11.922000000000001,"15min":7.9800000000000004},"max":{"1min":69.796000000000006,"5min":69.796000000000006,"15min":321.79500000000002},"last":16.318999999999999},{"interface":"front","average":{"1min":19.149999999999999,"5min":13.789999999999999,"15min":21.59},"min":{"1min":8.3620000000000001,"5min":7.9699999999999998,"15min":7.7229999999999999},"max":{"1min":65.344999999999999,"5min":65.344999999999999,"15min":282.54599999999999},"last":14.542999999999999}]},{"osd":7,"last update":"Sun Jan 21 05:05:30 2024","interfaces":[{"interface":"back","average":{"1min":24.565000000000001,"5min":19.756,"15min":26.001000000000001},"min":{"1min":16.887,"5min":13.028,"15min":8.8460000000000001},"max":{"1min":67.106999999999999,"5min":67.106999999999999,"15min":312.435},"last":17.847999999999999},{"interface":"front","average":{"1min":20.120000000000001,"5min":15.542,"15min":23.835000000000001},"min":{"1min":8.5540000000000003,"5min":8.2370000000000001,"15min":8.2370000000000001},"max":{"1min":97.379000000000005,"5min":97.379000000000005,"15min":256.11900000000003},"last":18.655000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509831,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74652,"kb_used_data":11540,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297188,"statfs":{"total":96636764160,"available":96560320512,"internally_reserved":0,"allocated":11816960,"data_stored":6611742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":11.103999999999999,"5min":9.2059999999999995,"15min":15.691000000000001},"min":{"1min":4.1760000000000002,"5min":4.1760000000000002,"15min":4.1760000000000002},"max":{"1min":63.862000000000002,"5min":63.862000000000002,"15min":209.96299999999999},"last":5.7149999999999999},{"interface":"front","average":{"1min":12.346,"5min":9.9589999999999996,"15min":16.739000000000001},"min":{"1min":5.7009999999999996,"5min":4.9560000000000004,"15min":4.7549999999999999},"max":{"1min":51.240000000000002,"5min":51.240000000000002,"15min":244.054},"last":7.2699999999999996}]},{"osd":2,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":25.413,"5min":13.148999999999999,"15min":17.792999999999999},"min":{"1min":8.1020000000000003,"5min":6.202,"15min":4.8819999999999997},"max":{"1min":218.80500000000001,"5min":218.80500000000001,"15min":221.54300000000001},"last":8.75},{"interface":"front","average":{"1min":24.704999999999998,"5min":15.188000000000001,"15min":20.256},"min":{"1min":10.317,"5min":6.9960000000000004,"15min":6.9960000000000004},"max":{"1min":144.12899999999999,"5min":144.12899999999999,"15min":219.34899999999999},"last":11.407}]},{"osd":3,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":14.996,"5min":13.903,"15min":21.224},"min":{"1min":7.3200000000000003,"5min":6.569,"15min":6.569},"max":{"1min":40.685000000000002,"5min":40.685000000000002,"15min":163.58799999999999},"last":12.925000000000001},{"interface":"front","average":{"1min":14.272,"5min":12.506,"15min":19.687999999999999},"min":{"1min":7.9630000000000001,"5min":7.0549999999999997,"15min":6.4809999999999999},"max":{"1min":55.024999999999999,"5min":55.024999999999999,"15min":244.178},"last":9.4979999999999993}]},{"osd":4,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":20.52,"5min":16.398,"15min":21.395},"min":{"1min":11.083,"5min":7.6369999999999996,"15min":7.6369999999999996},"max":{"1min":89.608000000000004,"5min":89.608000000000004,"15min":241.94800000000001},"last":13.584},{"interface":"front","average":{"1min":16.532,"5min":11.914,"15min":18.238},"min":{"1min":6.6500000000000004,"5min":6.6500000000000004,"15min":6.6500000000000004},"max":{"1min":112.226,"5min":112.226,"15min":225.60499999999999},"last":10.688000000000001}]},{"osd":5,"last update":"Sun Jan 21 05:05:29 2024","interfaces":[{"interface":"back","average":{"1min":26.631,"5min":16.741,"15min":22.994},"min":{"1min":9.1850000000000005,"5min":9.1850000000000005,"15min":6.8970000000000002},"max":{"1min":156.554,"5min":156.554,"15min":313.36599999999999},"last":14.741},{"interface":"front","average":{"1min":26.550000000000001,"5min":16.300999999999998,"15min":23.606000000000002},"min":{"1min":8.9369999999999994,"5min":8.9369999999999994,"15min":5.851},"max":{"1min":138.114,"5min":138.114,"15min":298.49599999999998},"last":12.352}]},{"osd":6,"last update":"Sun Jan 21 05:05:46 2024","interfaces":[{"interface":"back","average":{"1min":21.718,"5min":16.562999999999999,"15min":24.361999999999998},"min":{"1min":11.117000000000001,"5min":8.8949999999999996,"15min":8.8949999999999996},"max":{"1min":69.263999999999996,"5min":69.263999999999996,"15min":298.51999999999998},"last":15.329000000000001},{"interface":"front","average":{"1min":36.866,"5min":21.071999999999999,"15min":25.277000000000001},"min":{"1min":13.145,"5min":11.101000000000001,"15min":7.4880000000000004},"max":{"1min":293.34699999999998,"5min":293.34699999999998,"15min":293.34699999999998},"last":17.690999999999999}]},{"osd":7,"last update":"Sun Jan 21 05:05:17 2024","interfaces":[{"interface":"back","average":{"1min":21.521000000000001,"5min":16.407,"15min":24.297999999999998},"min":{"1min":11.332000000000001,"5min":9.8070000000000004,"15min":9.3230000000000004},"max":{"1min":107.875,"5min":107.875,"15min":214.24199999999999},"last":16.765000000000001},{"interface":"front","average":{"1min":20.009,"5min":17.219999999999999,"15min":24.443000000000001},"min":{"1min":9.5779999999999994,"5min":9.5779999999999994,"15min":9.4120000000000008},"max":{"1min":51.546999999999997,"5min":51.546999999999997,"15min":180.73099999999999},"last":14.129}]}]},{"osd":5,"up_from":13,"seq":55834575239,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74648,"kb_used_data":11536,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94297192,"statfs":{"total":96636764160,"available":96560324608,"internally_reserved":0,"allocated":11812864,"data_stored":6611737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":23.058,"5min":12.195,"15min":18.460000000000001},"min":{"1min":6.226,"5min":5.2300000000000004,"15min":4.601},"max":{"1min":154.13399999999999,"5min":154.13399999999999,"15min":377.50999999999999},"last":5.8810000000000002},{"interface":"front","average":{"1min":32.405999999999999,"5min":17.396999999999998,"15min":21.706},"min":{"1min":7.8029999999999999,"5min":5.8559999999999999,"15min":4.4359999999999999},"max":{"1min":193.01400000000001,"5min":193.01400000000001,"15min":224.58600000000001},"last":11.752000000000001}]},{"osd":1,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":24.754999999999999,"5min":14.738,"15min":22.327999999999999},"min":{"1min":6.165,"5min":5.6820000000000004,"15min":4.5380000000000003},"max":{"1min":160.279,"5min":160.279,"15min":368.79199999999997},"last":12.737},{"interface":"front","average":{"1min":28.187000000000001,"5min":13.547000000000001,"15min":19.158999999999999},"min":{"1min":9.9169999999999998,"5min":4.944,"15min":4.3170000000000002},"max":{"1min":156.38499999999999,"5min":156.38499999999999,"15min":225.21000000000001},"last":10.699}]},{"osd":2,"last update":"Sun Jan 21 05:05:26 2024","interfaces":[{"interface":"back","average":{"1min":28.533000000000001,"5min":14.004,"15min":20.603999999999999},"min":{"1min":7.7069999999999999,"5min":6.6420000000000003,"15min":5.8460000000000001},"max":{"1min":232.78200000000001,"5min":232.78200000000001,"15min":232.78200000000001},"last":16.564},{"interface":"front","average":{"1min":35.103000000000002,"5min":15.885999999999999,"15min":22.988},"min":{"1min":7.9740000000000002,"5min":6.4199999999999999,"15min":5.8289999999999997},"max":{"1min":288.49099999999999,"5min":288.49099999999999,"15min":309.24599999999998},"last":13.361000000000001}]},{"osd":3,"last update":"Sun Jan 21 05:05:45 2024","interfaces":[{"interface":"back","average":{"1min":52.302999999999997,"5min":22.747,"15min":27.129000000000001},"min":{"1min":7.7610000000000001,"5min":7.7610000000000001,"15min":6.7229999999999999},"max":{"1min":271.49000000000001,"5min":271.49000000000001,"15min":375.07799999999997},"last":7.5010000000000003},{"interface":"front","average":{"1min":41.609999999999999,"5min":17.966999999999999,"15min":23.789000000000001},"min":{"1min":8.0259999999999998,"5min":6.4589999999999996,"15min":6.4589999999999996},"max":{"1min":257.07600000000002,"5min":257.07600000000002,"15min":276.57100000000003},"last":17.376999999999999}]},{"osd":4,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":31.202999999999999,"5min":15.868,"15min":24.077000000000002},"min":{"1min":8.9039999999999999,"5min":6.3360000000000003,"15min":6.3360000000000003},"max":{"1min":184.767,"5min":184.767,"15min":272.274},"last":15.23},{"interface":"front","average":{"1min":33.640999999999998,"5min":20.071999999999999,"15min":28.475000000000001},"min":{"1min":8.8109999999999999,"5min":8.8109999999999999,"15min":6.8689999999999998},"max":{"1min":204.47499999999999,"5min":204.47499999999999,"15min":378.404},"last":6.827}]},{"osd":6,"last update":"Sun Jan 21 05:05:45 2024","interfaces":[{"interface":"back","average":{"1min":48.223999999999997,"5min":19.954999999999998,"15min":25.443999999999999},"min":{"1min":9.4209999999999994,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":305.68299999999999,"5min":305.68299999999999,"15min":305.68299999999999},"last":18.207000000000001},{"interface":"front","average":{"1min":62.576000000000001,"5min":25.863,"15min":32.338999999999999},"min":{"1min":12.919,"5min":10.442,"15min":9.6750000000000007},"max":{"1min":333.18700000000001,"5min":333.18700000000001,"15min":333.18700000000001},"last":18.988}]},{"osd":7,"last update":"Sun Jan 21 05:05:33 2024","interfaces":[{"interface":"back","average":{"1min":31.890000000000001,"5min":20.614999999999998,"15min":29.431999999999999},"min":{"1min":10.763,"5min":9.9640000000000004,"15min":8.032},"max":{"1min":202.90700000000001,"5min":202.90700000000001,"15min":486.358},"last":8.2170000000000005},{"interface":"front","average":{"1min":36.167000000000002,"5min":20.385999999999999,"15min":27.478999999999999},"min":{"1min":10.587999999999999,"5min":9.8740000000000006,"15min":9.1039999999999992},"max":{"1min":177.25800000000001,"5min":177.25800000000001,"15min":486.62900000000002},"last":19.922999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-21T05:06:16.174 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-21T05:06:16.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.272+0000 7f2ceb628640 10 monclient: get_monmap_and_config 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.272+0000 7f2ceb628640 10 monclient: build_initial_monmap 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.272+0000 7f2ceb628640 10 monclient: monmap: 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T05:06:16.273131+0000 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T05:06:16.273131+0000 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:0: v1:172.21.15.69:6789/0 mon.noname-a-legacy 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:1: v1:172.21.15.144:6789/0 mon.noname-c-legacy 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-b 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:3: v2:172.21.15.69:3300/0 mon.noname-a 2024-01-21T05:06:16.273 INFO:teuthology.orchestra.run.smithi069.stderr:4: v2:172.21.15.144:3300/0 mon.noname-c 2024-01-21T05:06:16.274 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:16.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.272+0000 7f2ceb628640 1 Processor -- start 2024-01-21T05:06:16.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.272+0000 7f2ceb628640 1 -- start start 2024-01-21T05:06:16.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.272+0000 7f2ceb628640 10 monclient: init 2024-01-21T05:06:16.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 10 monclient: _reopen_session rank -1 2024-01-21T05:06:16.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-01-21T05:06:16.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce40a8a40 0x7f2ce4145b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:16.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 10 monclient(hunting): picked mon.noname-b con 0x7f2ce40a8a40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ce40b2210 0x7f2ce413f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 10 monclient(hunting): picked mon.noname-c con 0x7f2ce40b2210 addr v2:172.21.15.144:3300/0 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ce40a8e20 0x7f2ce413f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 10 monclient(hunting): picked mon.noname-a con 0x7f2ce40a8e20 addr v2:172.21.15.69:3300/0 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 10 monclient(hunting): start opening mon connection 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce40aa7b0 con 0x7f2ce40a8e20 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 10 monclient(hunting): start opening mon connection 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 1 -- --> v2:172.21.15.144:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce413fc40 con 0x7f2ce40b2210 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 10 monclient(hunting): start opening mon connection 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce4145700 con 0x7f2ce40a8a40 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 10 monclient(hunting): _renew_subs 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.273+0000 7f2ceb628640 10 monclient(hunting): authenticate will time out at 3334.614820s 2024-01-21T05:06:16.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce93c4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce40a8a40 0x7f2ce4145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce8bc3640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ce40b2210 0x7f2ce413f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce9bc5640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ce40a8e20 0x7f2ce413f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce8bc3640 1 --2- >> v2:172.21.15.144:3300/0 conn(0x7f2ce40b2210 0x7f2ce413f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.69:43852/0 (socket says 172.21.15.69:43852) 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce8bc3640 1 -- 172.21.15.69:0/2541784208 learned_addr learned my addr 172.21.15.69:0/2541784208 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce8bc3640 10 monclient(hunting): get_auth_request con 0x7f2ce40b2210 auth_method 0 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce8bc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce8bc3640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce8bc3640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce9bc5640 10 monclient(hunting): get_auth_request con 0x7f2ce40a8e20 auth_method 0 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce9bc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce9bc5640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce9bc5640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce93c4640 10 monclient(hunting): get_auth_request con 0x7f2ce40a8a40 auth_method 0 2024-01-21T05:06:16.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce93c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:06:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce93c4640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:06:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.274+0000 7f2ce93c4640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:06:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.275+0000 7f2ce8bc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:06:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.275+0000 7f2ce8bc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:06:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.275+0000 7f2ce8bc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:06:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.275+0000 7f2ce9bc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:06:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.275+0000 7f2ce9bc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:06:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.275+0000 7f2ce9bc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:06:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.275+0000 7f2ce93c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:06:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.275+0000 7f2ce93c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:06:16.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.275+0000 7f2ce93c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:06:16.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.276+0000 7f2ce8bc3640 10 monclient(hunting): handle_auth_done global_id 21640 payload 274 2024-01-21T05:06:16.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.276+0000 7f2ce8bc3640 1 -- 172.21.15.69:0/2541784208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce40a8a40 msgr2=0x7f2ce4145b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:06:16.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.276+0000 7f2ce8bc3640 1 --2- 172.21.15.69:0/2541784208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce40a8a40 0x7f2ce4145b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.276+0000 7f2ce8bc3640 1 -- 172.21.15.69:0/2541784208 >> v2:172.21.15.69:3300/0 conn(0x7f2ce40a8e20 msgr2=0x7f2ce413f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-21T05:06:16.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.276+0000 7f2ce8bc3640 1 --2- 172.21.15.69:0/2541784208 >> v2:172.21.15.69:3300/0 conn(0x7f2ce40a8e20 0x7f2ce413f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.276+0000 7f2ce8bc3640 10 monclient: _finish_hunting 0 2024-01-21T05:06:16.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.276+0000 7f2ce8bc3640 1 monclient: found mon.noname-c 2024-01-21T05:06:16.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.276+0000 7f2ce8bc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:06:16.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.276+0000 7f2ce8bc3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.144:3300/0 2024-01-21T05:06:16.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.276+0000 7f2ce8bc3640 1 -- 172.21.15.69:0/2541784208 --> v2:172.21.15.144:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce40aa9b0 con 0x7f2ce40b2210 2024-01-21T05:06:16.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.278+0000 7f2ce8bc3640 1 --2- 172.21.15.69:0/2541784208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce40b2210 0x7f2ce413f1c0 secure :-1 s=READY pgs=5244 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009870 tx=0x7f2cd802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a37cd69a24001c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:06:16.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 1 -- 172.21.15.69:0/2541784208 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8031640 con 0x7f2ce40b2210 2024-01-21T05:06:16.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:06:16.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-01-21T05:06:16.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 10 monclient: dump: 2024-01-21T05:06:16.283 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:06:16.283 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:06:16.283 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:16.283 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:16.283 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:06:16.283 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:16.283 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 10 monclient: _finish_auth 0 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 10 monclient: _check_auth_tickets 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 1 -- 172.21.15.69:0/2541784208 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd80317a0 con 0x7f2ce40b2210 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 1 -- 172.21.15.69:0/2541784208 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8031b30 con 0x7f2ce40b2210 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ceb628640 5 monclient: authenticate success, global_id 21640 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ceb628640 10 monclient: get_monmap_and_config success 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ceb628640 4 set_mon_vals no callback set 2024-01-21T05:06:16.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ce3fff640 10 monclient: dump: 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:06:16.285 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:16.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ceb628640 10 monclient: shutdown 2024-01-21T05:06:16.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ceb628640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:06:16.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ceb628640 1 -- 172.21.15.69:0/2541784208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce40b2210 msgr2=0x7f2ce413f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:06:16.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.282+0000 7f2ceb628640 1 --2- 172.21.15.69:0/2541784208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce40b2210 0x7f2ce413f1c0 secure :-1 s=READY pgs=5244 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8009870 tx=0x7f2cd802b8f0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.283+0000 7f2ceb628640 1 -- 172.21.15.69:0/2541784208 shutdown_connections 2024-01-21T05:06:16.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.283+0000 7f2ceb628640 1 --2- 172.21.15.69:0/2541784208 >> v2:172.21.15.69:3300/0 conn(0x7f2ce40a8e20 0x7f2ce413f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.283+0000 7f2ceb628640 1 --2- 172.21.15.69:0/2541784208 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce40b2210 0x7f2ce413f1c0 unknown :-1 s=CLOSED pgs=5244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.283+0000 7f2ceb628640 1 --2- 172.21.15.69:0/2541784208 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce40a8a40 0x7f2ce4145b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.283+0000 7f2ceb628640 1 -- 172.21.15.69:0/2541784208 >> 172.21.15.69:0/2541784208 conn(0x7f2ce4066f10 msgr2=0x7f2ce40b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.283+0000 7f2ceb628640 1 -- 172.21.15.69:0/2541784208 shutdown_connections 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.283+0000 7f2ceb628640 1 -- 172.21.15.69:0/2541784208 wait complete. 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.283+0000 7f2ceb628640 10 monclient: build_initial_monmap 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.283+0000 7f2ceb628640 10 monclient: monmap: 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 0 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 0.000000 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:created 0.000000 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 0 (unknown) 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:16.289 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.noname-a 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.noname-b 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.noname-c 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 librados: starting msgr at 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 librados: starting objecter 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 Processor -- start 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 -- start start 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 librados: setting wanted keys 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 librados: calling monclient init 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 10 monclient: init 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 10 monclient: _reopen_session rank -1 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 10 monclient: _add_conns ranks=[2,1,0] 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce40a8a40 0x7f2ce40b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:16.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 10 monclient(hunting): picked mon.noname-c con 0x7f2ce40a8a40 addr [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce40a8e20 0x7f2ce413e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 10 monclient(hunting): picked mon.noname-b con 0x7f2ce40a8e20 addr [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce40b2210 0x7f2ce41dad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 10 monclient(hunting): picked mon.noname-a con 0x7f2ce40b2210 addr [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 10 monclient(hunting): start opening mon connection 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce41453b0 con 0x7f2ce40b2210 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 10 monclient(hunting): start opening mon connection 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce4144f40 con 0x7f2ce40a8e20 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 10 monclient(hunting): start opening mon connection 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 1 -- --> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce4145100 con 0x7f2ce40a8a40 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 10 monclient(hunting): _renew_subs 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.284+0000 7f2ceb628640 10 monclient(hunting): authenticate will time out at 3334.626002s 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce93c4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce40a8a40 0x7f2ce40b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:16.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce9bc5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce40b2210 0x7f2ce41dad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce8bc3640 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce40a8e20 0x7f2ce413e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce93c4640 1 --2- >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce40a8a40 0x7f2ce40b08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3301/0 says I am v2:172.21.15.69:39030/0 (socket says 172.21.15.69:39030) 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce93c4640 1 -- 172.21.15.69:0/2510991188 learned_addr learned my addr 172.21.15.69:0/2510991188 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce93c4640 10 monclient(hunting): get_auth_request con 0x7f2ce40a8a40 auth_method 0 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce93c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce93c4640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce93c4640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce9bc5640 10 monclient(hunting): get_auth_request con 0x7f2ce40b2210 auth_method 0 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce9bc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce9bc5640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce9bc5640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce8bc3640 10 monclient(hunting): get_auth_request con 0x7f2ce40a8e20 auth_method 0 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce8bc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-01-21T05:06:16.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce8bc3640 10 monclient(hunting): _init_auth method 2 2024-01-21T05:06:16.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.285+0000 7f2ce8bc3640 10 monclient(hunting): _init_auth creating new auth 2024-01-21T05:06:16.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.286+0000 7f2ce93c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:06:16.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.286+0000 7f2ce93c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:06:16.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.286+0000 7f2ce93c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:06:16.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.286+0000 7f2ce9bc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:06:16.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.286+0000 7f2ce9bc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:06:16.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.286+0000 7f2ce9bc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:06:16.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.286+0000 7f2ce8bc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-01-21T05:06:16.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.286+0000 7f2ce8bc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-01-21T05:06:16.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.286+0000 7f2ce8bc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-01-21T05:06:16.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.288+0000 7f2ce8bc3640 10 monclient(hunting): handle_auth_done global_id 21643 payload 824 2024-01-21T05:06:16.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.288+0000 7f2ce8bc3640 1 -- 172.21.15.69:0/2510991188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce40a8a40 msgr2=0x7f2ce40b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:06:16.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.288+0000 7f2ce8bc3640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce40a8a40 0x7f2ce40b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.288+0000 7f2ce8bc3640 1 -- 172.21.15.69:0/2510991188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce40b2210 msgr2=0x7f2ce41dad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:06:16.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.288+0000 7f2ce8bc3640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce40b2210 0x7f2ce41dad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.288+0000 7f2ce8bc3640 10 monclient: _finish_hunting 0 2024-01-21T05:06:16.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.288+0000 7f2ce8bc3640 1 monclient: found mon.noname-b 2024-01-21T05:06:16.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.288+0000 7f2ce8bc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-01-21T05:06:16.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.288+0000 7f2ce8bc3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.144:3300/0 2024-01-21T05:06:16.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.288+0000 7f2ce8bc3640 1 -- 172.21.15.69:0/2510991188 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd80094d0 con 0x7f2ce40a8e20 2024-01-21T05:06:16.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.290+0000 7f2ce8bc3640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce40a8e20 0x7f2ce413e2b0 secure :-1 s=READY pgs=5245 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8033000 tx=0x7f2cd80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:06:16.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.290+0000 7f2ce1ffb640 10 client.?.objecter ms_handle_connect 0x7f2ce40a8e20 2024-01-21T05:06:16.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.290+0000 7f2ce1ffb640 10 client.?.objecter resend_mon_ops 2024-01-21T05:06:16.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 1 -- 172.21.15.69:0/2510991188 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd803a930 con 0x7f2ce40a8e20 2024-01-21T05:06:16.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:06:16.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-01-21T05:06:16.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 10 monclient: dump: 2024-01-21T05:06:16.296 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:06:16.296 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:06:16.296 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 10 monclient: _finish_auth 0 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 10 monclient: _check_auth_tickets 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 1 -- 172.21.15.69:0/2510991188 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd803aa90 con 0x7f2ce40a8e20 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ceb628640 5 monclient: authenticate success, global_id 21643 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 10 monclient: handle_config config(0 keys) v1 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ceb628640 10 monclient: _renew_subs 2024-01-21T05:06:16.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 1 -- 172.21.15.69:0/2510991188 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd803ae40 con 0x7f2ce40a8e20 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ceb628640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ceb628640 1 -- 172.21.15.69:0/2510991188 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce40aad40 con 0x7f2ce40a8e20 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce37fe640 4 set_mon_vals no callback set 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ce1ffb640 10 monclient: dump: 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:epoch 1 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:fsid 396a1cf2-e196-4a31-b42f-998c792d93aa 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:last_changed 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:created 2024-01-21T04:31:59.217447+0000 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:min_mon_release 19 (squid) 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:election_strategy: 1 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-01-21T05:06:16.298 INFO:teuthology.orchestra.run.smithi069.stderr:1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.b 2024-01-21T05:06:16.299 INFO:teuthology.orchestra.run.smithi069.stderr:2: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-01-21T05:06:16.299 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:06:16.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ceb628640 10 client.21643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-01-21T05:06:16.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ceb628640 10 monclient: _renew_subs 2024-01-21T05:06:16.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ceb628640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:06:16.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ceb628640 1 -- 172.21.15.69:0/2510991188 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce40ac750 con 0x7f2ce40a8e20 2024-01-21T05:06:16.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.294+0000 7f2ceb628640 1 librados: init done 2024-01-21T05:06:16.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.295+0000 7f2ceb628640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-01-21T05:06:16.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.295+0000 7f2ceb628640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:06:16.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.295+0000 7f2ceb628640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:06:16.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.295+0000 7f2ceb628640 1 -- 172.21.15.69:0/2510991188 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce40aa1c0 con 0x7f2ce40a8e20 2024-01-21T05:06:16.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.536+0000 7f2ce1ffb640 1 -- 172.21.15.69:0/2510991188 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 5) v1 ==== 89198+0+0 (secure 0 0 0) 0x7f2cd802f050 con 0x7f2ce40a8e20 2024-01-21T05:06:16.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.537+0000 7f2ce1ffb640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2cd006c8b0 0x7f2cd006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-21T05:06:16.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.537+0000 7f2ce93c4640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2cd006c8b0 0x7f2cd006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-21T05:06:16.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.537+0000 7f2ce93c4640 10 monclient: get_auth_request con 0x7f2cd006c8b0 auth_method 0 2024-01-21T05:06:16.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.537+0000 7f2ce1ffb640 10 client.21643.objecter ms_handle_connect 0x7f2cd006c8b0 2024-01-21T05:06:16.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.537+0000 7f2ce93c4640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2cd006c8b0 0x7f2cd006ed70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009a40 tx=0x7f2cd400d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-21T05:06:16.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.539+0000 7f2ce1ffb640 1 -- 172.21.15.69:0/2510991188 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(1109..1109 src has 1059..1109) v4 ==== 6333+0+0 (secure 0 0 0) 0x7f2cd807a350 con 0x7f2ce40a8e20 2024-01-21T05:06:16.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.539+0000 7f2ce1ffb640 10 client.21643.objecter ms_dispatch 0x7f2ce4000b90 osd_map(1109..1109 src has 1059..1109) v4 2024-01-21T05:06:16.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.539+0000 7f2ce1ffb640 3 client.21643.objecter handle_osd_map got epochs [1109,1109] > 0 2024-01-21T05:06:16.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.539+0000 7f2ce1ffb640 3 client.21643.objecter handle_osd_map decoding full epoch 1109 2024-01-21T05:06:16.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.539+0000 7f2ce1ffb640 20 client.21643.objecter dump_active .. 0 homeless 2024-01-21T05:06:16.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.549+0000 7f2ce1ffb640 1 -- 172.21.15.69:0/2510991188 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180410 (secure 0 0 0) 0x7f2cd8076690 con 0x7f2ce40a8e20 2024-01-21T05:06:16.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.549+0000 7f2ce1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-01-21T05:06:16.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.549+0000 7f2ce1ffb640 10 monclient: _finish_command 1 = system:0 2024-01-21T05:06:16.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.726+0000 7f2ceb628640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-01-21T05:06:16.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.726+0000 7f2ceb628640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-01-21T05:06:16.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.726+0000 7f2ceb628640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.144:3300/0 2024-01-21T05:06:16.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.726+0000 7f2ceb628640 1 -- 172.21.15.69:0/2510991188 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f2ce40ac750 con 0x7f2ce40a8e20 2024-01-21T05:06:16.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.911+0000 7f2ce1ffb640 1 -- 172.21.15.69:0/2510991188 <== mon.1 v2:172.21.15.144:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2cd8076690 con 0x7f2ce40a8e20 2024-01-21T05:06:16.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.911+0000 7f2ce1ffb640 10 monclient: handle_config config(1 keys) v1 2024-01-21T05:06:16.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.911+0000 7f2ce37fe640 4 set_mon_vals no callback set 2024-01-21T05:06:16.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.911+0000 7f2ce37fe640 10 set_mon_vals mon_health_to_clog = false 2024-01-21T05:06:16.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.911+0000 7f2ce1ffb640 1 -- 172.21.15.69:0/2510991188 <== mon.1 v2:172.21.15.144:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v5)=0 v5) v1 ==== 123+0+0 (secure 0 0 0) 0x7f2ce40ac750 con 0x7f2ce40a8e20 2024-01-21T05:06:16.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.911+0000 7f2ce1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-01-21T05:06:16.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.911+0000 7f2ce1ffb640 10 monclient: _finish_command 2 = system:0 2024-01-21T05:06:16.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 10 librados: watch_flush enter 2024-01-21T05:06:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 10 librados: watch_flush exit 2024-01-21T05:06:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 20 client.21643.objecter shutdown clearing up homeless session... 2024-01-21T05:06:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 10 client.21643.objecter successfully canceled tick 2024-01-21T05:06:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 -- 172.21.15.69:0/2510991188 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2cd006c8b0 msgr2=0x7f2cd006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:06:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2cd006c8b0 0x7f2cd006ed70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009a40 tx=0x7f2cd400d040 comp rx=0 tx=0).stop 2024-01-21T05:06:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 10 monclient: shutdown 2024-01-21T05:06:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 20 monclient: shutdown discarding 0 pending message(s) 2024-01-21T05:06:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 -- 172.21.15.69:0/2510991188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce40a8e20 msgr2=0x7f2ce413e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-21T05:06:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce40a8e20 0x7f2ce413e2b0 secure :-1 s=READY pgs=5245 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8033000 tx=0x7f2cd80046c0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 -- 172.21.15.69:0/2510991188 shutdown_connections 2024-01-21T05:06:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.144:6800/1966288294,v1:172.21.15.144:6801/1966288294] conn(0x7f2cd006c8b0 0x7f2cd006ed70 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce40b2210 0x7f2ce41dad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2ce40a8e20 0x7f2ce413e2b0 unknown :-1 s=CLOSED pgs=5245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 --2- 172.21.15.69:0/2510991188 >> [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] conn(0x7f2ce40a8a40 0x7f2ce40b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-21T05:06:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 -- 172.21.15.69:0/2510991188 >> 172.21.15.69:0/2510991188 conn(0x7f2ce4066f10 msgr2=0x7f2ce40b2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-21T05:06:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 -- 172.21.15.69:0/2510991188 shutdown_connections 2024-01-21T05:06:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 -- 172.21.15.69:0/2510991188 wait complete. 2024-01-21T05:06:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-01-21T05:06:16.913+0000 7f2ceb628640 1 librados: shutdown 2024-01-21T05:06:16.925 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-21T05:06:16.925 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-21T05:06:16.925 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-21T05:06:16.925 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:06:16.938 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T05:06:16.928+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 34525) UID: 0 2024-01-21T05:06:16.938 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-01-21T05:06:16.930+0000 b47e640 -1 osd.0 1109 *** Got signal Terminated *** 2024-01-21T05:06:24.454 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:52.360 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:24.459 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:52.366 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:24.478 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:52.385 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:24.505 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:52.381 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:24.508 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:52.384 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:24.524 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:52.431 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:24.526 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:52.401 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:24.528 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:52.435 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:24.545 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:52.452 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:29.474 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:57.381 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:29.477 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:57.384 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:29.492 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:33:57.399 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:29.524 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:57.399 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:29.528 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:57.435 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:29.528 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:57.403 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:29.530 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:57.437 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:29.538 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:33:57.413 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:29.540 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:33:57.448 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:49.525 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:17.431 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:49.527 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:17.434 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:49.544 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:17.451 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:49.575 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:17.451 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:49.578 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:17.454 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:49.594 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:17.469 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:49.595 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:17.502 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:49.598 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:17.505 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:49.613 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:17.520 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:50.188 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:18.064 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:50.189 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:18.065 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:50.197 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:18.073 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:50.264 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:18.171 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:50.265 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:18.172 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:50.277 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:18.184 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:50.281 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:18.188 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:50.283 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:18.190 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:06:50.298 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:18.205 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:03.954 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:31.861 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:03.958 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:31.865 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:03.964 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:31.871 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:03.965 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:31.840 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:03.967 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:31.874 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:03.967 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:31.843 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:03.983 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:31.890 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:03.991 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:31.866 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:03.994 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:31.901 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:09.623 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:37.529 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:09.625 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:37.532 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:09.641 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:37.548 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:09.674 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:37.549 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:09.675 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:37.582 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:09.676 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:37.551 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:09.679 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:37.586 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:09.686 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:37.561 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:09.690 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:37.597 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:10.159 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:38.034 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:10.160 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:38.035 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:10.168 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:38.043 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:10.179 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:38.086 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:10.182 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:38.090 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:10.193 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:38.101 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:10.199 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:38.106 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:10.200 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:38.107 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:10.209 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:38.116 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:11.837 INFO:tasks.ceph.osd.0:Stopped 2024-01-21T05:07:11.837 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-21T05:07:11.837 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:07:11.853 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T05:07:11.840+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 34527) UID: 0 2024-01-21T05:07:11.853 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-01-21T05:07:11.842+0000 b47e640 -1 osd.1 1115 *** Got signal Terminated *** 2024-01-21T05:07:29.674 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:57.581 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:29.677 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:57.584 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:29.706 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:34:57.613 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:29.726 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:57.633 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:29.727 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:57.602 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:29.730 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:57.605 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:29.730 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:57.638 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:29.749 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:34:57.624 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:29.749 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:34:57.657 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:49.722 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:17.628 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:49.726 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:17.634 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:49.751 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:17.658 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:49.779 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:17.685 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:49.779 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:17.686 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:49.780 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:17.653 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:49.780 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:17.655 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:49.796 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:17.672 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:49.801 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:17.708 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:54.740 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:22.646 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:54.744 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:22.651 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:54.746 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:22.621 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:54.748 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:22.624 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:54.762 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:22.637 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:54.763 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:22.670 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:54.792 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:22.699 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:54.795 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:22.702 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:07:54.806 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:22.713 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:09.777 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:37.684 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:09.782 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:37.689 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:09.804 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:37.711 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:09.831 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:37.738 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:09.832 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:37.739 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:09.832 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:37.708 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:09.835 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:37.710 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:09.848 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:37.755 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:09.850 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:37.725 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:12.851 INFO:tasks.ceph.osd.1:Stopped 2024-01-21T05:08:12.851 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-21T05:08:12.851 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:08:12.867 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T05:08:12.855+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 34530) UID: 0 2024-01-21T05:08:12.867 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-01-21T05:08:12.856+0000 b47e640 -1 osd.2 1118 *** Got signal Terminated *** 2024-01-21T05:08:14.796 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:42.703 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:14.801 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:42.708 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:14.817 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:42.724 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:14.848 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:42.723 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:14.849 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:42.725 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:14.850 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:42.757 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:14.855 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:42.762 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:14.863 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:42.738 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:14.870 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:42.777 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:15.339 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:43.243 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:15.339 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:43.245 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:15.346 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:35:43.253 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:15.363 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:43.238 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:15.364 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:43.239 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:15.372 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:35:43.247 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:15.377 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:43.284 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:15.377 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:43.285 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:15.386 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:35:43.293 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:17.000 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-21T05:08:18.953 INFO:tasks.ceph.osd.2:Stopped 2024-01-21T05:08:18.953 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-21T05:08:18.953 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:08:18.967 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T05:08:18.957+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 34532) UID: 0 2024-01-21T05:08:18.967 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-01-21T05:08:18.958+0000 b47e640 -1 osd.3 1121 *** Got signal Terminated *** 2024-01-21T05:08:22.907 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-01-21T05:08:22.907 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-01-21T05:08:25.055 INFO:tasks.ceph.osd.3:Stopped 2024-01-21T05:08:25.055 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-21T05:08:25.055 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:08:25.068 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T05:08:25.059+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 34556) UID: 0 2024-01-21T05:08:25.071 INFO:tasks.ceph.osd.4.smithi144.stderr:2024-01-21T05:08:25.060+0000 b47e640 -1 osd.4 1123 *** Got signal Terminated *** 2024-01-21T05:08:28.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-01-21T05:08:28.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-01-21T05:08:31.157 INFO:tasks.ceph.osd.4:Stopped 2024-01-21T05:08:31.157 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-21T05:08:31.157 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:08:31.171 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T05:08:31.161+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 34561) UID: 0 2024-01-21T05:08:31.171 INFO:tasks.ceph.osd.5.smithi144.stderr:2024-01-21T05:08:31.163+0000 b47e640 -1 osd.5 1125 *** Got signal Terminated *** 2024-01-21T05:08:34.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-01-21T05:08:34.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-01-21T05:08:37.259 INFO:tasks.ceph.osd.5:Stopped 2024-01-21T05:08:37.259 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-21T05:08:37.259 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:08:37.272 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T05:08:37.263+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 34560) UID: 0 2024-01-21T05:08:37.274 INFO:tasks.ceph.osd.6.smithi144.stderr:2024-01-21T05:08:37.264+0000 b47e640 -1 osd.6 1127 *** Got signal Terminated *** 2024-01-21T05:08:39.895 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:36:07.801 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:39.899 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:36:07.806 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:39.929 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:36:07.836 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:39.944 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:36:07.851 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:39.947 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:36:07.854 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:39.949 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:36:07.824 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:39.952 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:36:07.827 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:39.975 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:36:07.882 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:39.979 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:36:07.854 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:40.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-01-21T05:08:40.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-01-21T05:08:40.391 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:36:08.266 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:40.392 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:36:08.267 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:40.405 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:36:08.280 34023== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:40.427 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:36:08.335 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:40.429 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:36:08.336 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:40.429 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:36:08.336 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:40.430 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:36:08.337 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:40.438 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:36:08.345 34243== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:40.441 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:36:08.348 34237== Warning: unimplemented fcntl command: 1036 2024-01-21T05:08:43.360 INFO:tasks.ceph.osd.6:Stopped 2024-01-21T05:08:43.360 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-21T05:08:43.361 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:08:43.377 INFO:tasks.ceph.osd.7.smithi144.stderr:2024-01-21T05:08:43.364+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 34564) UID: 0 2024-01-21T05:08:43.377 INFO:tasks.ceph.osd.7.smithi144.stderr:2024-01-21T05:08:43.366+0000 b47e640 -1 osd.7 1127 *** Got signal Terminated *** 2024-01-21T05:08:45.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-01-21T05:08:45.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-01-21T05:08:49.462 INFO:tasks.ceph.osd.7:Stopped 2024-01-21T05:08:49.463 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-21T05:08:49.463 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-21T05:08:49.463 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:08:49.486 INFO:tasks.ceph.mgr.y:Stopped 2024-01-21T05:08:49.487 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-21T05:08:49.487 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:08:49.488 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T05:08:49.487+0000 7fc4f6533640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34021) UID: 0 2024-01-21T05:08:49.488 INFO:tasks.ceph.mgr.x.smithi144.stderr:2024-01-21T05:08:49.487+0000 7fc4f6533640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-01-21T05:08:49.516 INFO:tasks.ceph.mgr.x:Stopped 2024-01-21T05:08:49.516 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-21T05:08:49.516 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-21T05:08:49.516 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:08:49.541 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-01-21T05:08:49.530+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34209) UID: 0 2024-01-21T05:08:49.542 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-01-21T05:08:49.531+0000 9f77640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-01-21T05:08:50.371 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:36:18.278 34237== 2024-01-21T05:08:50.371 INFO:tasks.ceph.mon.a.smithi069.stderr:==00:00:36:18.278 34237== Exit program on first error (--exit-on-first-error=yes) 2024-01-21T05:08:50.393 INFO:tasks.ceph.mon.a.smithi069.stderr:daemon-helper: command failed with exit status 42 2024-01-21T05:08:50.729 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-01-21T05:08:50.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-01-21T05:08:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-21T05:08:55.619 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-01-21T05:08:55.620 INFO:tasks.ceph.mon.a:Stopped 2024-01-21T05:08:55.620 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-21T05:08:55.620 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:08:55.636 INFO:tasks.ceph.mon.c.smithi069.stderr:2024-01-21T05:08:55.624+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34211) UID: 0 2024-01-21T05:08:55.637 INFO:tasks.ceph.mon.c.smithi069.stderr:2024-01-21T05:08:55.625+0000 9f77640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-01-21T05:08:55.932 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-01-21T05:08:56.262 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:36:24.169 34243== 2024-01-21T05:08:56.262 INFO:tasks.ceph.mon.c.smithi069.stderr:==00:00:36:24.169 34243== Exit program on first error (--exit-on-first-error=yes) 2024-01-21T05:08:56.283 INFO:tasks.ceph.mon.c.smithi069.stderr:daemon-helper: command failed with exit status 42 2024-01-21T05:09:01.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-01-21T05:09:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-21T05:09:01.722 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi069 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-01-21T05:09:01.723 INFO:tasks.ceph.mon.c:Stopped 2024-01-21T05:09:01.723 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-21T05:09:01.723 INFO:teuthology.orchestra.run:waiting for 300 2024-01-21T05:09:01.738 INFO:tasks.ceph.mon.b.smithi144.stderr:2024-01-21T05:09:01.727+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34019) UID: 0 2024-01-21T05:09:01.739 INFO:tasks.ceph.mon.b.smithi144.stderr:2024-01-21T05:09:01.728+0000 9f77640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-01-21T05:09:02.548 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:36:30.423 34023== 2024-01-21T05:09:02.548 INFO:tasks.ceph.mon.b.smithi144.stderr:==00:00:36:30.423 34023== Exit program on first error (--exit-on-first-error=yes) 2024-01-21T05:09:02.566 INFO:tasks.ceph.mon.b.smithi144.stderr:daemon-helper: command failed with exit status 42 2024-01-21T05:09:06.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-01-21T05:09:06.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-01-21T05:09:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-01-21T05:09:07.825 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi144 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-01-21T05:09:07.825 INFO:tasks.ceph.mon.b:Stopped 2024-01-21T05:09:07.825 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-21T05:09:07.826 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-01-21T05:09:07.949 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-21T05:09:07.949 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-21T05:09:08.060 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-21T05:09:08.061 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-21T05:09:08.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-21T05:09:08.139 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-21T05:09:08.216 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi069.front.sepia.ceph.com 2024-01-21T05:09:08.216 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-21T05:09:08.289 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi144.front.sepia.ceph.com 2024-01-21T05:09:08.289 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-21T05:09:08.451 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi144.front.sepia.ceph.com 2024-01-21T05:09:08.452 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-21T05:09:08.538 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi144.front.sepia.ceph.com 2024-01-21T05:09:08.538 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-21T05:09:08.595 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi144.front.sepia.ceph.com 2024-01-21T05:09:08.595 DEBUG:teuthology.orchestra.run.smithi144:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-21T05:09:08.662 INFO:tasks.ceph:Archiving mon data... 2024-01-21T05:09:08.663 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711/data/mon.a.tgz 2024-01-21T05:09:08.663 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-01-21T05:09:08.679 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.Lf1BnA0aFZ 2024-01-21T05:09:08.680 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Lf1BnA0aFZ 2024-01-21T05:09:08.840 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.Lf1BnA0aFZ 2024-01-21T05:09:08.927 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.Lf1BnA0aFZ is 457KB 2024-01-21T05:09:09.025 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.Lf1BnA0aFZ 2024-01-21T05:09:09.039 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711/data/mon.c.tgz 2024-01-21T05:09:09.039 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-01-21T05:09:09.093 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.w9alyY4JDM 2024-01-21T05:09:09.093 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.w9alyY4JDM 2024-01-21T05:09:09.236 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.w9alyY4JDM 2024-01-21T05:09:09.320 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.w9alyY4JDM is 457KB 2024-01-21T05:09:09.416 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.w9alyY4JDM 2024-01-21T05:09:09.430 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711/data/mon.b.tgz 2024-01-21T05:09:09.430 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2024-01-21T05:09:09.448 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.QOpkgB7Sv2 2024-01-21T05:09:09.448 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.QOpkgB7Sv2 2024-01-21T05:09:09.604 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.QOpkgB7Sv2 2024-01-21T05:09:09.691 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.QOpkgB7Sv2 is 457KB 2024-01-21T05:09:09.788 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.QOpkgB7Sv2 2024-01-21T05:09:09.803 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-21T05:09:09.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-21T05:09:09.805 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-21T05:09:10.123 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-21T05:09:10.359 INFO:tasks.ceph:Archiving crash dumps... 2024-01-21T05:09:10.360 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711/remote/smithi069/crash 2024-01-21T05:09:10.361 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-21T05:09:10.388 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711/remote/smithi144/crash 2024-01-21T05:09:10.389 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-21T05:09:10.415 INFO:tasks.ceph:Compressing logs... 2024-01-21T05:09:10.415 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T05:09:10.432 DEBUG:teuthology.orchestra.run.smithi144:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-21T05:09:10.455 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107845.log 2024-01-21T05:09:10.456 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34251.log 2024-01-21T05:09:10.471 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107803.log 2024-01-21T05:09:10.471 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.34251.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-21T05:09:10.471 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107803.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107845.log.gz 2024-01-21T05:09:10.471 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84282.log 2024-01-21T05:09:10.471 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70008.log 2024-01-21T05:09:10.471 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107803.log.gz 2024-01-21T05:09:10.471 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.84282.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.34251.log.gz 2024-01-21T05:09:10.471 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40615.log 2024-01-21T05:09:10.471 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40379.log 2024-01-21T05:09:10.471 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40955.log 2024-01-21T05:09:10.471 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38601.log 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69667.log 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40370.log 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69667.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38601.log.gz 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.84282.log.gz 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108139.log 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% 90.7% -- replaced with /var/log/ceph/ceph-client.admin.70008.log.gz 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40370.log: -- replaced with /var/log/ceph/ceph-client.admin.69667.log.gz 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40219.log 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.108139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37875.log 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40219.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40615.log.gz 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39317.log 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108139.log.gz 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.37875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107299.log 2024-01-21T05:09:10.472 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39317.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37875.log.gz 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.107719.log 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.39317.log.gz 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107719.log: /var/log/ceph/ceph-client.admin.107299.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/mon.c.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107719.log.gz 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.107299.log.gz 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose --/var/log/ceph/valgrind/osd.1.log: /var/log/ceph/valgrind/osd.3.log 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/mon.a.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38947.log 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-01-21T05:09:10.473 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/valgrind/osd.3.log: 64.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39559.log 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39935.log 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.40219.log.gz 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38147.log 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.40379.log.gz 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39559.log: /var/log/ceph/ceph-client.admin.39935.log: -- replaced with /var/log/ceph/ceph-client.admin.38947.log.gz 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41052.log 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41042.log 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107971.log 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39559.log.gz 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38147.log.gz 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.40370.log.gz 2024-01-21T05:09:10.474 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41042.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.39935.log.gz 2024-01-21T05:09:10.486 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38425.log 2024-01-21T05:09:10.486 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39517.log 2024-01-21T05:09:10.486 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34720.log 2024-01-21T05:09:10.486 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107971.log.gz/var/log/ceph/ceph-client.admin.39517.log: 2024-01-21T05:09:10.486 INFO:teuthology.orchestra.run.smithi069.stderr:gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38425.log.gz 2024-01-21T05:09:10.486 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38383.log 2024-01-21T05:09:10.486 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.34720.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39517.log.gz 2024-01-21T05:09:10.486 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41029.log 2024-01-21T05:09:10.486 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.38383.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41077.log 2024-01-21T05:09:10.486 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34720.log.gz/var/log/ceph/ceph-client.admin.41029.log: 2024-01-21T05:09:10.487 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38383.log.gz 2024-01-21T05:09:10.487 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34557.log 2024-01-21T05:09:10.487 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107887.log 2024-01-21T05:09:10.487 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.34557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40909.log 2024-01-21T05:09:10.487 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107887.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34557.log.gz 2024-01-21T05:09:10.487 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40122.log 2024-01-21T05:09:10.487 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40909.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107887.log.gz 2024-01-21T05:09:10.487 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39883.log 2024-01-21T05:09:10.494 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34467.log 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.34467.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34256.log 2024-01-21T05:09:10.495 INFO:teuthology.orchestra.run.smithi144.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34467.log.gz 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.34256.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34256.log.gz 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34336.log 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34418.log 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.34336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34176.log 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.34418.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34336.log.gz 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34296.log 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.34176.log: 87.3%/var/log/ceph/ceph-mgr.x.log: -- replaced with /var/log/ceph/ceph-client.admin.34418.log.gz 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34176.log.gz 2024-01-21T05:09:10.496 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34376.log 2024-01-21T05:09:10.497 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-21T05:09:10.497 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.34296.log: /var/log/ceph/ceph-client.admin.34376.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34296.log.gz 2024-01-21T05:09:10.497 INFO:teuthology.orchestra.run.smithi144.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34376.log.gz 2024-01-21T05:09:10.497 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-21T05:09:10.497 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34216.log 2024-01-21T05:09:10.497 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 92.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-21T05:09:10.498 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40122.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40955.log.gz 2024-01-21T05:09:10.498 INFO:teuthology.orchestra.run.smithi069.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.41077.log.gz 2024-01-21T05:09:10.498 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.41029.log.gz 2024-01-21T05:09:10.498 INFO:teuthology.orchestra.run.smithi069.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.admin.40909.log.gz 2024-01-21T05:09:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84278.log 2024-01-21T05:09:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39928.log 2024-01-21T05:09:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.84278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107467.log 2024-01-21T05:09:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39219.log 2024-01-21T05:09:10.500 INFO:teuthology.orchestra.run.smithi144.stderr: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-21T05:09:10.500 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/ceph-client.admin.34216.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34216.log.gz 2024-01-21T05:09:10.501 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107467.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107467.log.gz 2024-01-21T05:09:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40734.log 2024-01-21T05:09:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39219.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39219.log.gz 2024-01-21T05:09:10.502 INFO:teuthology.orchestra.run.smithi069.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.84278.log.gz 2024-01-21T05:09:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60085.log 2024-01-21T05:09:10.502 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39279.log 2024-01-21T05:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40734.log: /var/log/ceph/ceph-client.admin.60085.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.60085.log.gz 2024-01-21T05:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.39883.log.gz 2024-01-21T05:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38299.log 2024-01-21T05:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40758.log 2024-01-21T05:09:10.507 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38299.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39279.log.gz 2024-01-21T05:09:10.507 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38299.log.gz 2024-01-21T05:09:10.507 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35565.log 2024-01-21T05:09:10.507 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39408.log 2024-01-21T05:09:10.508 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.35565.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35565.log.gz 2024-01-21T05:09:10.509 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.40734.log.gz 2024-01-21T05:09:10.510 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107425.log 2024-01-21T05:09:10.510 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40549.log 2024-01-21T05:09:10.510 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39408.log: /var/log/ceph/ceph-client.admin.107425.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39408.log.gz 2024-01-21T05:09:10.511 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-21T05:09:10.511 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107425.log.gz 2024-01-21T05:09:10.515 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40549.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40758.log.gz 2024-01-21T05:09:10.515 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40829.log 2024-01-21T05:09:10.515 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40549.log.gz 2024-01-21T05:09:10.515 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38597.log 2024-01-21T05:09:10.515 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60086.log 2024-01-21T05:09:10.515 INFO:teuthology.orchestra.run.smithi069.stderr: 96.7% -- replaced with /var/log/ceph/ceph-client.admin.41042.log.gz 2024-01-21T05:09:10.515 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38597.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38597.log.gz 2024-01-21T05:09:10.515 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107635.log 2024-01-21T05:09:10.515 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41055.log 2024-01-21T05:09:10.516 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107635.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107635.log.gz 2024-01-21T05:09:10.516 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98163.log 2024-01-21T05:09:10.517 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41055.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.60086.log.gz 2024-01-21T05:09:10.519 INFO:teuthology.orchestra.run.smithi069.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41055.log.gz 2024-01-21T05:09:10.520 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40164.log 2024-01-21T05:09:10.520 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107341.log 2024-01-21T05:09:10.524 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.98163.log: /var/log/ceph/ceph-client.admin.40164.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.98163.log.gz 2024-01-21T05:09:10.524 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40415.log 2024-01-21T05:09:10.527 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107341.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.40829.log.gz 2024-01-21T05:09:10.527 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107341.log.gz 2024-01-21T05:09:10.527 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38868.log 2024-01-21T05:09:10.527 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38881.log 2024-01-21T05:09:10.527 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38868.log: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-21T05:09:10.527 INFO:teuthology.orchestra.run.smithi069.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38868.log.gz 2024-01-21T05:09:10.527 INFO:teuthology.orchestra.run.smithi069.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.41052.log.gz 2024-01-21T05:09:10.527 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39960.log 2024-01-21T05:09:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38341.log 2024-01-21T05:09:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38604.log 2024-01-21T05:09:10.535 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38881.log.gz 2024-01-21T05:09:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38600.log 2024-01-21T05:09:10.535 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38604.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40164.log.gz 2024-01-21T05:09:10.535 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38341.log.gz 2024-01-21T05:09:10.535 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38604.log.gz 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38607.log 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39350.log 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38607.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38600.log.gz 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39350.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38607.log.gz 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39350.log.gz 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40402.log 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38856.log 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107677.log 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.39960.log.gz 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40131.log 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107677.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38856.log.gz 2024-01-21T05:09:10.536 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38795.log 2024-01-21T05:09:10.537 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107677.log.gz 2024-01-21T05:09:10.537 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107761.log 2024-01-21T05:09:10.537 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38795.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38795.log.gz 2024-01-21T05:09:10.539 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70009.log 2024-01-21T05:09:10.539 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107761.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107761.log.gz 2024-01-21T05:09:10.539 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40324.log 2024-01-21T05:09:10.539 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70009.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70009.log.gz 2024-01-21T05:09:10.539 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36765.log 2024-01-21T05:09:10.541 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40324.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40402.log.gz 2024-01-21T05:09:10.547 INFO:teuthology.orchestra.run.smithi069.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.40415.log.gz 2024-01-21T05:09:10.547 INFO:teuthology.orchestra.run.smithi069.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40324.log.gz 2024-01-21T05:09:10.547 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38203.log 2024-01-21T05:09:10.551 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.36765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40702.log 2024-01-21T05:09:10.551 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40744.log 2024-01-21T05:09:10.551 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36765.log.gz 2024-01-21T05:09:10.551 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40702.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38203.log.gz 2024-01-21T05:09:10.551 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107509.log 2024-01-21T05:09:10.551 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40744.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28398.log 2024-01-21T05:09:10.551 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107509.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.40702.log.gz 2024-01-21T05:09:10.551 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107509.log.gz 2024-01-21T05:09:10.551 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70010.log 2024-01-21T05:09:10.552 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107257.log 2024-01-21T05:09:10.552 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.tmp-client.admin.28398.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28398.log.gz 2024-01-21T05:09:10.552 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108055.log 2024-01-21T05:09:10.553 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107257.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107257.log.gz 2024-01-21T05:09:10.554 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.40744.log.gz 2024-01-21T05:09:10.554 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-21T05:09:10.554 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40442.log 2024-01-21T05:09:10.555 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.108055.log: /var/log/ceph/ceph-osd.3.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.70010.log.gz 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108055.log.gz 2024-01-21T05:09:10.555 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:09:10.555 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40741.log 2024-01-21T05:09:10.555 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39475.log 2024-01-21T05:09:10.557 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40442.log: /var/log/ceph/ceph-client.admin.40741.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.40741.log.gz 2024-01-21T05:09:10.557 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40442.log.gz 2024-01-21T05:09:10.558 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38259.log 2024-01-21T05:09:10.558 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38941.log 2024-01-21T05:09:10.558 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39475.log: /var/log/ceph/ceph-client.admin.38259.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39475.log.gz 2024-01-21T05:09:10.559 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.38259.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.38082.log 2024-01-21T05:09:10.559 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:09:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38941.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.39928.log.gz 2024-01-21T05:09:10.562 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38941.log.gz 2024-01-21T05:09:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107551.log 2024-01-21T05:09:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38606.log 2024-01-21T05:09:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2024-01-21T05:09:10.562 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38082.log.gz 2024-01-21T05:09:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38606.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39059.log 2024-01-21T05:09:10.562 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.107551.log.gz 2024-01-21T05:09:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40284.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38606.log.gz 2024-01-21T05:09:10.562 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107929.log 2024-01-21T05:09:10.564 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69666.log 2024-01-21T05:09:10.564 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39059.log: /var/log/ceph/ceph-client.admin.107929.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39059.log.gz 2024-01-21T05:09:10.564 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38861.log 2024-01-21T05:09:10.564 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107929.log.gz 2024-01-21T05:09:10.566 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69666.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2024-01-21T05:09:10.566 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84281.log 2024-01-21T05:09:10.569 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38861.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.69666.log.gz 2024-01-21T05:09:10.569 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73747.log 2024-01-21T05:09:10.569 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.84281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107383.log 2024-01-21T05:09:10.570 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38861.log.gz 2024-01-21T05:09:10.570 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98162.log 2024-01-21T05:09:10.570 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107383.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.107383.log.gz 2024-01-21T05:09:10.570 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39139.log 2024-01-21T05:09:10.571 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.98162.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.84281.log.gz 91.1% -- replaced with /var/log/ceph/ceph-client.admin.73747.log.gz 2024-01-21T05:09:10.571 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:09:10.571 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-01-21T05:09:10.571 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84280.log 2024-01-21T05:09:10.572 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39139.log: /var/log/ceph/ceph-mgr.y.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39139.log.gz 2024-01-21T05:09:10.573 INFO:teuthology.orchestra.run.smithi069.stderr: 90.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40484.log 2024-01-21T05:09:10.573 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.98162.log.gz 2024-01-21T05:09:10.573 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40157.log 2024-01-21T05:09:10.576 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.84280.log: /var/log/ceph/ceph-client.admin.40484.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.84280.log.gz 2024-01-21T05:09:10.576 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40507.log 2024-01-21T05:09:10.577 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40157.log: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.40122.log.gz 2024-01-21T05:09:10.578 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41074.log 2024-01-21T05:09:10.581 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40507.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.40507.log.gz 2024-01-21T05:09:10.581 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60087.log 2024-01-21T05:09:10.582 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41074.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40157.log.gz 2024-01-21T05:09:10.583 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108179.log 2024-01-21T05:09:10.584 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60087.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.41074.log.gz 2024-01-21T05:09:10.584 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73746.log 2024-01-21T05:09:10.585 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.108179.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108179.log.gz 2024-01-21T05:09:10.585 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40192.log 2024-01-21T05:09:10.586 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73746.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.60087.log.gz 2024-01-21T05:09:10.586 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-21T05:09:10.589 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40192.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.73746.log.gz 2024-01-21T05:09:10.589 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39844.log 2024-01-21T05:09:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.0.log: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-21T05:09:10.600 INFO:teuthology.orchestra.run.smithi069.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.40192.log.gz 2024-01-21T05:09:10.600 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84279.log 2024-01-21T05:09:10.612 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98161.log 2024-01-21T05:09:10.615 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.84279.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.84279.log.gz 2024-01-21T05:09:10.626 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-21T05:09:10.626 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.98161.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40484.log.gz 2024-01-21T05:09:10.627 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40700.log 2024-01-21T05:09:10.629 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.98161.log.gz 2024-01-21T05:09:10.630 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40568.log 2024-01-21T05:09:10.635 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40700.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40700.log.gz 2024-01-21T05:09:10.635 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73748.log 2024-01-21T05:09:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40568.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39844.log.gz 2024-01-21T05:09:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38602.log 2024-01-21T05:09:10.639 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73748.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.73748.log.gz 2024-01-21T05:09:10.640 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-21T05:09:10.641 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38602.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38602.log.gz 2024-01-21T05:09:10.641 INFO:teuthology.orchestra.run.smithi069.stderr: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-21T05:09:10.641 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39867.log 2024-01-21T05:09:10.642 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40033.log 2024-01-21T05:09:10.644 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.39867.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.40568.log.gz 2024-01-21T05:09:10.654 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39179.log 2024-01-21T05:09:10.667 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39922.log 2024-01-21T05:09:10.669 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39179.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39179.log.gz 2024-01-21T05:09:10.681 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39240.log 2024-01-21T05:09:10.694 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39922.log: 92.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108097.log 2024-01-21T05:09:10.696 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39240.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39240.log.gz 2024-01-21T05:09:10.702 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107593.log 2024-01-21T05:09:10.702 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.108097.log: -- replaced with /var/log/ceph/ceph-client.admin.39922.log.gz 2024-01-21T05:09:10.712 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% 91.8% -- replaced with /var/log/ceph/ceph-client.admin.39867.log.gz 2024-01-21T05:09:10.712 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.108097.log.gz 2024-01-21T05:09:10.713 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38605.log 2024-01-21T05:09:10.714 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.107593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108013.log 2024-01-21T05:09:10.715 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38605.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107593.log.gz 2024-01-21T05:09:10.716 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38605.log.gz 2024-01-21T05:09:10.716 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39940.log 2024-01-21T05:09:10.717 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.108013.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108013.log.gz 2024-01-21T05:09:10.717 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40761.log 2024-01-21T05:09:10.721 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39940.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.39940.log.gz 2024-01-21T05:09:10.721 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40614.log 2024-01-21T05:09:10.722 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40761.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40761.log.gz 2024-01-21T05:09:10.723 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34580.log 2024-01-21T05:09:10.729 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40614.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40614.log.gz 2024-01-21T05:09:10.729 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40903.log 2024-01-21T05:09:10.730 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.34580.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34580.log.gz 2024-01-21T05:09:10.731 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-21T05:09:10.772 INFO:teuthology.orchestra.run.smithi144.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-21T05:09:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40903.log: /var/log/ceph/ceph-mon.c.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.40903.log.gz 2024-01-21T05:09:11.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-01-21T05:09:11.064 INFO:teuthology.orchestra.run.smithi069.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.40131.log.gz 2024-01-21T05:09:12.060 INFO:teuthology.orchestra.run.smithi069.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40033.log.gz 2024-01-21T05:09:12.248 INFO:teuthology.orchestra.run.smithi144.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-21T05:09:12.535 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-21T05:09:14.199 INFO:teuthology.orchestra.run.smithi069.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-21T05:09:16.459 INFO:teuthology.orchestra.run.smithi144.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-21T05:09:16.850 INFO:teuthology.orchestra.run.smithi144.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-21T05:09:16.989 INFO:teuthology.orchestra.run.smithi069.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-21T05:09:17.206 INFO:teuthology.orchestra.run.smithi069.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-21T05:09:17.542 INFO:teuthology.orchestra.run.smithi144.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-21T05:09:17.563 INFO:teuthology.orchestra.run.smithi144.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-21T05:09:17.565 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-01-21T05:09:17.565 INFO:teuthology.orchestra.run.smithi144.stderr:real 0m7.096s 2024-01-21T05:09:17.565 INFO:teuthology.orchestra.run.smithi144.stderr:user 0m27.625s 2024-01-21T05:09:17.566 INFO:teuthology.orchestra.run.smithi144.stderr:sys 0m0.908s 2024-01-21T05:09:18.213 INFO:teuthology.orchestra.run.smithi069.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-21T05:09:18.312 INFO:teuthology.orchestra.run.smithi069.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-21T05:09:18.314 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-01-21T05:09:18.314 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m7.870s 2024-01-21T05:09:18.314 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m35.440s 2024-01-21T05:09:18.314 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m1.285s 2024-01-21T05:09:18.314 INFO:tasks.ceph:Archiving logs... 2024-01-21T05:09:18.314 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711/remote/smithi069/log 2024-01-21T05:09:18.315 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T05:09:23.191 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711/remote/smithi144/log 2024-01-21T05:09:23.191 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-21T05:09:27.093 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-21T05:09:27.095 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-21T05:09:27.103 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0edf41a622739843d6b978b179ff3227b476dd9d/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-21T05:09:27.104 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-21T05:09:27.104 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T05:09:27.107 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-21T05:09:27.196 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-21T05:09:27.204 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-21T05:09:27.204 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T05:09:27.206 DEBUG:teuthology.orchestra.run.smithi144:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-21T05:09:27.220 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-01-21T05:09:27.223 INFO:teuthology.orchestra.run.smithi144.stderr:bash: line 1: ntpq: command not found 2024-01-21T05:09:27.224 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T05:09:27.224 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-01-21T05:09:27.224 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +104us[ +84us] +/- 87ms 2024-01-21T05:09:27.224 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -838us[ -858us] +/- 75ms 2024-01-21T05:09:27.225 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +773us[ +754us] +/- 76ms 2024-01-21T05:09:27.225 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T05:09:27.227 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-21T05:09:27.227 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2024-01-21T05:09:27.227 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +108us[ +103us] +/- 87ms 2024-01-21T05:09:27.227 INFO:teuthology.orchestra.run.smithi144.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -834us[ -839us] +/- 76ms 2024-01-21T05:09:27.227 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +786us[ +781us] +/- 76ms 2024-01-21T05:09:27.227 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-21T05:09:27.228 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-21T05:09:27.236 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-21T05:09:27.237 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-21T05:09:27.245 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T05:09:27.338 DEBUG:teuthology.orchestra.run.smithi144:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-21T05:09:27.407 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T05:09:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T05:09:27.438 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-21T05:09:27.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T05:09:27.469 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-21T05:09:27.479 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-21T05:09:27.487 INFO:teuthology.task.internal:Duration was 3162.703950 seconds 2024-01-21T05:09:27.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-21T05:09:27.496 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-21T05:09:27.496 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T05:09:27.499 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-21T05:09:27.540 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T05:09:27.554 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-21T05:09:27.837 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-21T05:09:27.837 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-01-21T05:09:27.837 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T05:09:27.857 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi144.front.sepia.ceph.com 2024-01-21T05:09:27.857 DEBUG:teuthology.orchestra.run.smithi144:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-21T05:09:27.898 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-21T05:09:27.898 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T05:09:27.901 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-21T05:09:27.967 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-21T05:09:27.967 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T05:09:27.970 DEBUG:teuthology.orchestra.run.smithi144:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-21T05:09:28.106 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-21T05:09:28.116 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-21T05:09:28.117 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T05:09:28.143 DEBUG:teuthology.orchestra.run.smithi144:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-21T05:09:28.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-21T05:09:28.179 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T05:09:28.186 DEBUG:teuthology.orchestra.run.smithi144:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-21T05:09:28.208 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-01-21T05:09:28.235 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = core 2024-01-21T05:09:28.253 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T05:09:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T05:09:28.281 DEBUG:teuthology.orchestra.run.smithi144:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-21T05:09:28.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-21T05:09:28.308 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-21T05:09:28.318 INFO:teuthology.task.internal:Transferring archived files... 2024-01-21T05:09:28.319 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711/remote/smithi069 2024-01-21T05:09:28.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T05:09:28.377 DEBUG:teuthology.misc:Transferring archived files from smithi144:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711/remote/smithi144 2024-01-21T05:09:28.377 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-21T05:09:28.412 INFO:teuthology.task.internal:Removing archive directory... 2024-01-21T05:09:28.412 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T05:09:28.414 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-21T05:09:28.461 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-21T05:09:28.473 INFO:teuthology.task.internal:Not uploading archives. 2024-01-21T05:09:28.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-21T05:09:28.483 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-21T05:09:28.483 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T05:09:28.485 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-21T05:09:28.497 INFO:teuthology.orchestra.run.smithi069.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 05:09 /home/ubuntu/cephtest 2024-01-21T05:09:28.515 INFO:teuthology.orchestra.run.smithi144.stdout: 265818 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 21 05:09 /home/ubuntu/cephtest 2024-01-21T05:09:28.519 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-21T05:09:28.529 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-21T05:09:28.540 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-01-21T05:09:28.540 INFO:teuthology.nuke:Checking targets against current locks 2024-01-21T05:09:28.564 DEBUG:teuthology.nuke:shortname: smithi069 2024-01-21T05:09:28.565 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T05:09:28.590 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 04:07:59.658104', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJHhHBSinmQEofBQijDsym1K+SkxvxwWk9w04OcLVikHqZ6Uhn4Og3fI/aWrXrLEWArQdCj/i62a0r94q3wei3Q='} 2024-01-21T05:09:28.593 DEBUG:teuthology.nuke:shortname: smithi144 2024-01-21T05:09:28.594 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-21T05:09:28.617 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-19_16:35:08-rados-wip-yuri8-testing-2024-01-18-0823-distro-default-smithi/7523711', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-21 04:07:59.659951', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPS4pNmIUfUvxeA5qS72DYwNe/zOtn6/GnN+cuB4nt/rM9D0AzxuESufov7R1JEN2b2pqSV503uetyIVzM9KDoA='} 2024-01-21T05:09:28.647 INFO:teuthology.orchestra.console.smithi069:Power off 2024-01-21T05:09:28.647 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T05:09:28.665 INFO:teuthology.orchestra.console.smithi144:Power off 2024-01-21T05:09:28.665 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-21T05:09:28.674 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-01-21T05:09:28.674 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T05:09:28.687 DEBUG:teuthology.orchestra.console.smithi144:power off output: Chassis Power Control: Down/Off 2024-01-21T05:09:28.687 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T05:09:28.696 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-01-21T05:09:28.709 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2024-01-21T05:09:32.697 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T05:09:32.710 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T05:09:32.806 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-01-21T05:09:32.818 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2024-01-21T05:09:36.808 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T05:09:36.819 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T05:09:36.918 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-01-21T05:09:36.927 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is on 2024-01-21T05:09:37.020 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-01-21T05:09:40.931 DEBUG:teuthology.orchestra.console.smithi144:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-21T05:09:41.042 DEBUG:teuthology.orchestra.console.smithi144:check power output: Chassis Power is off 2024-01-21T05:09:41.143 INFO:teuthology.orchestra.console.smithi144:Power off completed 2024-01-21T05:09:41.247 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} duration: 3162.703950405121 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-01-21T05:09:41.247 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-21T05:09:41.326 INFO:teuthology.run:FAIL